[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:43:36 dist-test-slave-4hd3 krb5kdc[400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:43:36 dist-test-slave-4hd3 krb5kdc[400](info): set up 2 sockets
Sep 26 20:43:36 dist-test-slave-4hd3 krb5kdc[400](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 20:43:37.768447 393 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 20:43:37.946909 393 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.178s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 20:43:38.123622 393 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.176s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260926 20:43:38.248787 393 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.125s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:43:38 dist-test-slave-4hd3 krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:43:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:43:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:43:38.307284 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:44803
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:44803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:38.387111 413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:38.387346 413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:38.387382 413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:38.388510 413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:43:38.388581 413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:38.388614 413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:38.388643 413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:43:38.388671 413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:43:38.390390 413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:44803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:44803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:38.390686 413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:38.390935 413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:38.395951 419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:38.396000 421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:38.395972 413 server_base.cc:1034] running on GCE node
W20260926 20:43:38.395961 418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:38.396390 413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:38.396718 413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:38.397887 413 hybrid_clock.cc:648] HybridClock initialized: now 1790455418397870 us; error 29 us; skew 500 ppm
Sep 26 20:43:38 dist-test-slave-4hd3 krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:38.404212 413 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:43:38.404701 413 webserver.cc:460] Webserver started at http://127.0.98.126:39897/ using document root <none> and password file <none>
I20260926 20:43:38.404932 413 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:38.405000 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:38.405170 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:38.406256 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "01f53a1b20e946c4ae2592c28704f958"
format_stamp: "Formatted at 2026-09-26 20:43:38 on dist-test-slave-4hd3"
server_key: "77bad5c33f919a7ec55f91a2da019563"
server_key_iv: "464aa992e7b037e99e27c200db0a51bf"
server_key_version: "encryptionkey@0"
I20260926 20:43:38.406638 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "01f53a1b20e946c4ae2592c28704f958"
format_stamp: "Formatted at 2026-09-26 20:43:38 on dist-test-slave-4hd3"
server_key: "77bad5c33f919a7ec55f91a2da019563"
server_key_iv: "464aa992e7b037e99e27c200db0a51bf"
server_key_version: "encryptionkey@0"
I20260926 20:43:38.408048 413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:43:38.408758 428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:38.408919 413 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:43:38.409034 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "01f53a1b20e946c4ae2592c28704f958"
format_stamp: "Formatted at 2026-09-26 20:43:38 on dist-test-slave-4hd3"
server_key: "77bad5c33f919a7ec55f91a2da019563"
server_key_iv: "464aa992e7b037e99e27c200db0a51bf"
server_key_version: "encryptionkey@0"
I20260926 20:43:38.409107 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:38.429869 413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:38.431965 413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:38.432160 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:38.436534 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:44803
I20260926 20:43:38.436586 480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:44803 every 8 connection(s)
I20260926 20:43:38.436971 413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:43:38.437815 481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:38.440076 481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958: Bootstrap starting.
I20260926 20:43:38.440661 481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:38.440951 481 log.cc:824] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:38.441641 481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958: No bootstrap required, opened a new log
I20260926 20:43:38.442229 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 413
I20260926 20:43:38.442345 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:43:38.443127 393 external_mini_cluster.cc:1428] Setting key 5d90ffe915bbb054ef75bb88f02bbf49
I20260926 20:43:38.443991 481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01f53a1b20e946c4ae2592c28704f958" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44803 } }
I20260926 20:43:38.444165 481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:38.444229 481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01f53a1b20e946c4ae2592c28704f958, State: Initialized, Role: FOLLOWER
I20260926 20:43:38.444406 481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01f53a1b20e946c4ae2592c28704f958" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44803 } }
I20260926 20:43:38.444504 481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:43:38.444556 481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:43:38.444603 481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:38.445415 481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01f53a1b20e946c4ae2592c28704f958" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44803 } }
I20260926 20:43:38.445546 481 leader_election.cc:302] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01f53a1b20e946c4ae2592c28704f958; no voters:
I20260926 20:43:38.445715 481 leader_election.cc:288] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:43:38.445749 486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:38.445852 486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958 [term 1 LEADER]: Becoming Leader. State: Replica: 01f53a1b20e946c4ae2592c28704f958, State: Running, Role: LEADER
I20260926 20:43:38.445930 486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01f53a1b20e946c4ae2592c28704f958" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44803 } }
I20260926 20:43:38.445981 481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:43:38.446274 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01f53a1b20e946c4ae2592c28704f958. Latest consensus state: current_term: 1 leader_uuid: "01f53a1b20e946c4ae2592c28704f958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01f53a1b20e946c4ae2592c28704f958" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44803 } } }
I20260926 20:43:38.446354 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958 [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:38.446281 488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01f53a1b20e946c4ae2592c28704f958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01f53a1b20e946c4ae2592c28704f958" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44803 } } }
I20260926 20:43:38.446529 488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958 [sys.catalog]: This master's current role is: LEADER
W20260926 20:43:38.447808 501 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:43:38.447865 501 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:43:38.447942 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:43:38.448105 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:43:38.449682 492 catalog_manager.cc:1347] Generated new cluster ID: 62020fa2088f445c8207c4094f67e355
I20260926 20:43:38.449739 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:43:38 dist-test-slave-4hd3 krb5kdc[400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:38.461167 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:38.444100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37530 (local address 127.0.98.126:44803)
0926 20:43:38.444319 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:38.444328 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:38.445955 (+ 1627us) server_negotiation.cc:407] Connection header received
0926 20:43:38.446924 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:38.446931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:38.446993 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:38.447782 (+ 789us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:38.449269 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:38.450651 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:38.452063 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:38.452686 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:38.452811 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:38.457142 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:38.457146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:38.457146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:38.457158 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:38.459150 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:38.459636 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:38.459639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:38.459640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:38.459778 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:38.460059 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:38.460060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:38.460061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:38.460363 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:38.460490 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:38.460790 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 20:43:38.460929 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":104,"threads_started":1}
I20260926 20:43:38.463104 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:43:38.463593 492 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:43:38.466391 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01f53a1b20e946c4ae2592c28704f958: Generated new TSK 0
I20260926 20:43:38.466552 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260926 20:43:38.494665 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:44803
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:38.574977 507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:38.575229 507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:38.575270 507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:38.576540 507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:38.576611 507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:38.576656 507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:43:38.578132 507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:44803
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:38.578398 507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:38.578635 507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:38.583643 513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:38.583693 512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:38.583654 515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:38.583874 507 server_base.cc:1034] running on GCE node
I20260926 20:43:38.584093 507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:38.584342 507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:38.585500 507 hybrid_clock.cc:648] HybridClock initialized: now 1790455418585469 us; error 60 us; skew 500 ppm
Sep 26 20:43:38 dist-test-slave-4hd3 krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:38.588100 507 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:43:38.588541 507 webserver.cc:460] Webserver started at http://127.0.98.65:36303/ using document root <none> and password file <none>
I20260926 20:43:38.588730 507 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:38.588771 507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:38.588860 507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:38.589886 507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "eb93d4e0f0714c7eab85feb9a01bae20"
format_stamp: "Formatted at 2026-09-26 20:43:38 on dist-test-slave-4hd3"
server_key: "18675679a4bad47200bc1a73d67bf1e5"
server_key_iv: "3280b7a4e65795b878150566499fa426"
server_key_version: "encryptionkey@0"
I20260926 20:43:38.590209 507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "eb93d4e0f0714c7eab85feb9a01bae20"
format_stamp: "Formatted at 2026-09-26 20:43:38 on dist-test-slave-4hd3"
server_key: "18675679a4bad47200bc1a73d67bf1e5"
server_key_iv: "3280b7a4e65795b878150566499fa426"
server_key_version: "encryptionkey@0"
I20260926 20:43:38.591507 507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:43:38.592334 522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:38.592569 507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:43:38.592675 507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "eb93d4e0f0714c7eab85feb9a01bae20"
format_stamp: "Formatted at 2026-09-26 20:43:38 on dist-test-slave-4hd3"
server_key: "18675679a4bad47200bc1a73d67bf1e5"
server_key_iv: "3280b7a4e65795b878150566499fa426"
server_key_version: "encryptionkey@0"
I20260926 20:43:38.592751 507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:38.606976 507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:38.609380 507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:38.609543 507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:38.609767 507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:38.610070 507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:38.610128 507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:38.610169 507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:38.610198 507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:38.616343 635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:44167 every 8 connection(s)
I20260926 20:43:38.616343 507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:44167
I20260926 20:43:38.616789 507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:43:38.619498 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 507
I20260926 20:43:38.619572 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:43:38.619764 393 external_mini_cluster.cc:1428] Setting key 324d7c538e90fe582a963059fc51dbcf
Sep 26 20:43:38 dist-test-slave-4hd3 krb5kdc[400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:38.627274 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:38.617568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:35573 (local address 127.0.98.126:44803)
0926 20:43:38.617610 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:38.617612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:38.617849 (+ 237us) server_negotiation.cc:407] Connection header received
0926 20:43:38.618378 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:38.618379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:38.618396 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:38.618450 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:38.619961 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:38.620576 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:38.621607 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:38.621812 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:38.621899 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:38.624441 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:38.624442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:38.624443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:38.624449 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:38.625839 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:38.626314 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:38.626314 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:38.626315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:38.626362 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:38.626653 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:38.626653 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:38.626653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:38.626752 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:38.626800 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:38.627137 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 20:43:38.627211 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:43:38.627626 638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:38.617542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:44803 (local address 127.0.98.65:35573)
0926 20:43:38.617731 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:38.617765 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:38.618264 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:38.618527 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:38.618531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:38.619094 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:38.619282 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:38.619287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:38.620672 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:38.620673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:38.621483 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:38.621485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:38.621921 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:38.621922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:38.622042 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:38.622593 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:38.622601 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:38.624351 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:38.625926 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:38.625934 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:38.625937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:38.626235 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:38.626427 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:38.626428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:38.626428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:38.626583 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:38.626818 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:38.626828 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:38.626993 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 20:43:38.627121 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 20:43:38.627288 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":91,"threads_started":1}
I20260926 20:43:38.628660 636 heartbeater.cc:347] Connected to a master server at 127.0.98.126:44803
I20260926 20:43:38.628777 636 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260926 20:43:38.628999 636 heartbeater.cc:511] Master 127.0.98.126:44803 requested a full tablet report, sending...
I20260926 20:43:38.629539 445 ts_manager.cc:194] Registered new tserver with Master: eb93d4e0f0714c7eab85feb9a01bae20 (127.0.98.65:44167)
I20260926 20:43:38.630216 445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:35573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260926 20:43:38.646775 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:44803
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:38.725698 641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:38.725936 641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:38.725975 641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:38.727100 641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:38.727167 641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:38.727212 641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:43:38.728711 641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:44803
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:38.728979 641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:38.729221 641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:38.734297 649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:38.734386 641 server_base.cc:1034] running on GCE node
W20260926 20:43:38.734335 647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:38.734330 646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:38.734726 641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:38.734979 641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:38.736145 641 hybrid_clock.cc:648] HybridClock initialized: now 1790455418736147 us; error 45 us; skew 500 ppm
Sep 26 20:43:38 dist-test-slave-4hd3 krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:38.738778 641 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:43:38.739295 641 webserver.cc:460] Webserver started at http://127.0.98.66:33269/ using document root <none> and password file <none>
I20260926 20:43:38.739522 641 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:38.739593 641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:38.739794 641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:38.740887 641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "5b7e0f466915489e881a2ca505b37375"
format_stamp: "Formatted at 2026-09-26 20:43:38 on dist-test-slave-4hd3"
server_key: "d22285c8811880a74e6877928ffb69a8"
server_key_iv: "780108834809788f488c2a8e24ace2a8"
server_key_version: "encryptionkey@0"
I20260926 20:43:38.741267 641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "5b7e0f466915489e881a2ca505b37375"
format_stamp: "Formatted at 2026-09-26 20:43:38 on dist-test-slave-4hd3"
server_key: "d22285c8811880a74e6877928ffb69a8"
server_key_iv: "780108834809788f488c2a8e24ace2a8"
server_key_version: "encryptionkey@0"
I20260926 20:43:38.742686 641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:43:38.743474 656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:38.743649 641 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:43:38.743764 641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "5b7e0f466915489e881a2ca505b37375"
format_stamp: "Formatted at 2026-09-26 20:43:38 on dist-test-slave-4hd3"
server_key: "d22285c8811880a74e6877928ffb69a8"
server_key_iv: "780108834809788f488c2a8e24ace2a8"
server_key_version: "encryptionkey@0"
I20260926 20:43:38.743854 641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:38.756726 641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:38.758680 641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:38.758849 641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:38.759086 641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:38.759394 641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:38.759449 641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:38.759490 641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:38.759516 641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:38.765509 641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:38433
I20260926 20:43:38.765564 769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:38433 every 8 connection(s)
I20260926 20:43:38.765861 641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:43:38.771507 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 641
I20260926 20:43:38.771620 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:43:38.771822 393 external_mini_cluster.cc:1428] Setting key f808afe2ab32aa8d64425db8a5d14382
Sep 26 20:43:38 dist-test-slave-4hd3 krb5kdc[400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:38.775352 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:38.766628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:48913 (local address 127.0.98.126:44803)
0926 20:43:38.766717 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:38.766719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:38.766770 (+ 51us) server_negotiation.cc:407] Connection header received
0926 20:43:38.767261 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:38.767262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:38.767272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:38.767323 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:38.768304 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:38.768804 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:38.769805 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:38.769997 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:38.770066 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:38.772402 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:38.772403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:38.772404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:38.772412 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:38.773930 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:38.774361 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:38.774362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:38.774363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:38.774426 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:38.774693 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:38.774694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:38.774694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:38.774808 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:38.774884 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:38.775170 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 20:43:38.775253 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:43:38.775549 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:38.766507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:44803 (local address 127.0.98.66:48913)
0926 20:43:38.766658 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:38.766706 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:38.767179 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:38.767378 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:38.767382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:38.768021 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:38.768216 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:38.768221 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:38.768887 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:38.768888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:38.769719 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:38.769721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:38.770082 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:38.770083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:38.770177 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:38.770656 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:38.770664 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:38.772300 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:38.774042 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:38.774050 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:38.774052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:38.774281 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:38.774494 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:38.774495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:38.774495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:38.774615 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:38.774875 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:38.774882 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:38.775031 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 20:43:38.775138 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 20:43:38.775300 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260926 20:43:38.776130 770 heartbeater.cc:347] Connected to a master server at 127.0.98.126:44803
I20260926 20:43:38.776235 770 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:38.776443 770 heartbeater.cc:511] Master 127.0.98.126:44803 requested a full tablet report, sending...
I20260926 20:43:38.776819 445 ts_manager.cc:194] Registered new tserver with Master: 5b7e0f466915489e881a2ca505b37375 (127.0.98.66:38433)
I20260926 20:43:38.777202 445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:48913
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260926 20:43:38.796747 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:44803
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:38.878079 775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:38.878362 775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:38.878400 775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:38.879884 775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:38.879923 775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:38.879951 775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:43:38.881392 775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:44803
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:38.881632 775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:38.881876 775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:38.886694 780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:38.886722 783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:38.886700 781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:38.886761 775 server_base.cc:1034] running on GCE node
I20260926 20:43:38.887122 775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:38.887383 775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:38.888554 775 hybrid_clock.cc:648] HybridClock initialized: now 1790455418888541 us; error 49 us; skew 500 ppm
Sep 26 20:43:38 dist-test-slave-4hd3 krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:38.890964 775 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:43:38.891396 775 webserver.cc:460] Webserver started at http://127.0.98.67:45105/ using document root <none> and password file <none>
I20260926 20:43:38.891631 775 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:38.891701 775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:38.891882 775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:38.892904 775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "752a3281e6d54ac799bfa0020105853d"
format_stamp: "Formatted at 2026-09-26 20:43:38 on dist-test-slave-4hd3"
server_key: "6ccd0e51bf6ad5d87186fefad43f41ef"
server_key_iv: "28195b3b2e0f8347a7b4f390ce6eb922"
server_key_version: "encryptionkey@0"
I20260926 20:43:38.893275 775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "752a3281e6d54ac799bfa0020105853d"
format_stamp: "Formatted at 2026-09-26 20:43:38 on dist-test-slave-4hd3"
server_key: "6ccd0e51bf6ad5d87186fefad43f41ef"
server_key_iv: "28195b3b2e0f8347a7b4f390ce6eb922"
server_key_version: "encryptionkey@0"
I20260926 20:43:38.894580 775 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:43:38.895318 790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:38.895484 775 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:43:38.895586 775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "752a3281e6d54ac799bfa0020105853d"
format_stamp: "Formatted at 2026-09-26 20:43:38 on dist-test-slave-4hd3"
server_key: "6ccd0e51bf6ad5d87186fefad43f41ef"
server_key_iv: "28195b3b2e0f8347a7b4f390ce6eb922"
server_key_version: "encryptionkey@0"
I20260926 20:43:38.895681 775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:38.921079 775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:38.923031 775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:38.923194 775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:38.923439 775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:38.923805 775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:38.923857 775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:38.923899 775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:38.923926 775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:38.930524 775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:46227
I20260926 20:43:38.930563 903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:46227 every 8 connection(s)
I20260926 20:43:38.930919 775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:43:38 dist-test-slave-4hd3 krb5kdc[400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:38.940615 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:38.931876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:49301 (local address 127.0.98.126:44803)
0926 20:43:38.931944 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:38.931946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:38.932184 (+ 238us) server_negotiation.cc:407] Connection header received
0926 20:43:38.932701 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:38.932703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:38.932716 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:38.932770 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:38.933805 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:38.934307 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:38.935318 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:38.935548 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:38.935637 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:38.937958 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:38.937960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:38.937960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:38.937976 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:38.939145 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:38.939574 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:38.939575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:38.939576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:38.939662 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:38.939947 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:38.939948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:38.939948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:38.940069 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:38.940151 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:38.940429 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 20:43:38.940513 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:43:38.940817 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:38.931886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:44803 (local address 127.0.98.67:49301)
0926 20:43:38.932073 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:38.932107 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:38.932597 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:38.932857 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:38.932861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:38.933509 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:38.933703 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:38.933708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:38.934414 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:38.934416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:38.935224 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:38.935226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:38.935646 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:38.935648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:38.935829 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:38.936309 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:38.936317 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:38.937839 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:38.939241 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:38.939248 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:38.939250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:38.939481 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:38.939766 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:38.939766 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:38.939767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:38.939877 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:38.940142 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:38.940147 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:38.940292 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 20:43:38.940395 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 20:43:38.940560 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":120,"threads_started":1}
I20260926 20:43:38.941084 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 775
I20260926 20:43:38.941190 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:43:38.941412 393 external_mini_cluster.cc:1428] Setting key 46e7247b9540fff25bacd4d0fe156bc5
I20260926 20:43:38.941543 904 heartbeater.cc:347] Connected to a master server at 127.0.98.126:44803
I20260926 20:43:38.941679 904 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:38.941885 904 heartbeater.cc:511] Master 127.0.98.126:44803 requested a full tablet report, sending...
I20260926 20:43:38.942330 445 ts_manager.cc:194] Registered new tserver with Master: 752a3281e6d54ac799bfa0020105853d (127.0.98.67:46227)
I20260926 20:43:38.942814 445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:49301
I20260926 20:43:38.942909 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:43:38 dist-test-slave-4hd3 krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455418, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:43:38 dist-test-slave-4hd3 krb5kdc[400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455418, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260926 20:43:38.965175 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:38.957725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.0.98.65:44167)
0926 20:43:38.957898 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:38.957903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:38.957912 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:43:38.957939 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:38.957940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:38.957954 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:38.958019 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:38.958576 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:38.959307 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:38.960095 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:38.960404 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:38.960464 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:38.962537 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:38.962544 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:38.962544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:38.962551 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:38.964019 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:38.964387 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:38.964390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:38.964390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:38.964452 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:38.964639 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:38.964639 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:38.964640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:38.964770 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:38.964875 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:38.964938 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 20:43:38.965013 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
W20260926 20:43:38.965662 550 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48212
Sep 26 20:43:38 dist-test-slave-4hd3 krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:43:38 dist-test-slave-4hd3 krb5kdc[400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260926 20:43:38.987587 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:38.980993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48214 (local address 127.0.98.65:44167)
0926 20:43:38.981060 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:38.981062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:38.981088 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:43:38.981150 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:38.981151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:38.981162 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:38.981210 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:38.981774 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:38.982250 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:38.982940 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:38.983129 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:38.983176 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:38.985385 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:38.985387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:38.985387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:38.985394 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:38.986688 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:38.986985 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:38.986985 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:38.986986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:38.987035 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:38.987201 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:38.987201 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:38.987202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:38.987286 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:38.987372 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:38.987441 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 20:43:38.987497 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:43:38.988176 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 507
I20260926 20:43:38.993309 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 641
I20260926 20:43:38.999010 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 775
I20260926 20:43:39.003769 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 413
2026-09-26T20:43:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2386 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:43:39 dist-test-slave-4hd3 krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:43:39 dist-test-slave-4hd3 krb5kdc[918](info): set up 2 sockets
Sep 26 20:43:39 dist-test-slave-4hd3 krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260926 20:43:40.035143 393 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:43:40 dist-test-slave-4hd3 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:43:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:43:40.148612 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:35625
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:35625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:40.228142 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:40.228369 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:40.228404 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:40.229548 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:43:40.229619 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:40.229640 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:40.229660 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:43:40.229691 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:43:40.231178 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:35625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:35625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:40.231509 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:40.231696 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:40.236392 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:40.236387 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:40.236402 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:40.236691 931 server_base.cc:1034] running on GCE node
I20260926 20:43:40.236927 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:40.237157 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:40.238304 931 hybrid_clock.cc:648] HybridClock initialized: now 1790455420238281 us; error 43 us; skew 500 ppm
Sep 26 20:43:40 dist-test-slave-4hd3 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:40.240785 931 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:43:40.241187 931 webserver.cc:460] Webserver started at http://127.0.98.126:33325/ using document root <none> and password file <none>
I20260926 20:43:40.241452 931 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:40.241535 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:40.241693 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:40.242734 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "e940c5cdfe8c423383b2d49ff8ab2bb9"
format_stamp: "Formatted at 2026-09-26 20:43:40 on dist-test-slave-4hd3"
server_key: "f90b715dbc6de7b8842834d9f5ad39ab"
server_key_iv: "fb35a5c919183ed50629254352b09b6c"
server_key_version: "encryptionkey@0"
I20260926 20:43:40.243098 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "e940c5cdfe8c423383b2d49ff8ab2bb9"
format_stamp: "Formatted at 2026-09-26 20:43:40 on dist-test-slave-4hd3"
server_key: "f90b715dbc6de7b8842834d9f5ad39ab"
server_key_iv: "fb35a5c919183ed50629254352b09b6c"
server_key_version: "encryptionkey@0"
I20260926 20:43:40.244462 931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:43:40.245182 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:40.245340 931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:43:40.245435 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "e940c5cdfe8c423383b2d49ff8ab2bb9"
format_stamp: "Formatted at 2026-09-26 20:43:40 on dist-test-slave-4hd3"
server_key: "f90b715dbc6de7b8842834d9f5ad39ab"
server_key_iv: "fb35a5c919183ed50629254352b09b6c"
server_key_version: "encryptionkey@0"
I20260926 20:43:40.245517 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:40.263010 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:40.264943 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:40.265120 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:40.268821 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:35625
I20260926 20:43:40.268857 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:35625 every 8 connection(s)
I20260926 20:43:40.269232 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:43:40.270159 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:40.272549 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9: Bootstrap starting.
I20260926 20:43:40.272811 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 931
I20260926 20:43:40.272910 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:43:40.273088 393 external_mini_cluster.cc:1428] Setting key d3215b779647cd92ae021ef3df871381
I20260926 20:43:40.273270 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:40.273608 999 log.cc:824] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:40.274281 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9: No bootstrap required, opened a new log
I20260926 20:43:40.276197 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e940c5cdfe8c423383b2d49ff8ab2bb9" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35625 } }
I20260926 20:43:40.276357 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:40.276405 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e940c5cdfe8c423383b2d49ff8ab2bb9, State: Initialized, Role: FOLLOWER
I20260926 20:43:40.276569 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e940c5cdfe8c423383b2d49ff8ab2bb9" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35625 } }
I20260926 20:43:40.276664 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:43:40.276710 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:43:40.276757 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:40.277305 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e940c5cdfe8c423383b2d49ff8ab2bb9" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35625 } }
I20260926 20:43:40.277424 999 leader_election.cc:302] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e940c5cdfe8c423383b2d49ff8ab2bb9; no voters:
I20260926 20:43:40.277650 999 leader_election.cc:288] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:43:40.277683 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:40.277887 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9 [term 1 LEADER]: Becoming Leader. State: Replica: e940c5cdfe8c423383b2d49ff8ab2bb9, State: Running, Role: LEADER
I20260926 20:43:40.278005 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:43:40.278381 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e940c5cdfe8c423383b2d49ff8ab2bb9" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35625 } }
I20260926 20:43:40.278759 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e940c5cdfe8c423383b2d49ff8ab2bb9. Latest consensus state: current_term: 1 leader_uuid: "e940c5cdfe8c423383b2d49ff8ab2bb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e940c5cdfe8c423383b2d49ff8ab2bb9" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35625 } } }
I20260926 20:43:40.279353 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:40.279093 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e940c5cdfe8c423383b2d49ff8ab2bb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e940c5cdfe8c423383b2d49ff8ab2bb9" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35625 } } }
I20260926 20:43:40.279582 1017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:43:40.279588 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9 [sys.catalog]: This master's current role is: LEADER
Sep 26 20:43:40 dist-test-slave-4hd3 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:40.280200 1017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:43:40.281823 1017 catalog_manager.cc:1347] Generated new cluster ID: 1393321b11c146e79311c63136caf8a5
I20260926 20:43:40.281884 1017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:43:40.286381 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:40.273635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56690 (local address 127.0.98.126:35625)
0926 20:43:40.273809 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:40.273820 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:40.273853 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:43:40.274285 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:40.274291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:40.274389 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:40.274567 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:40.275175 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.276139 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.277088 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.277591 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.277721 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:40.280599 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:40.280603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:40.280603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:40.280614 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:40.284673 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:40.285166 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:40.285169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:40.285169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:40.285271 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:40.285494 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:40.285495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:40.285495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:40.285774 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:40.285919 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:40.286062 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 20:43:40.286186 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260926 20:43:40.298647 1017 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:43:40.299371 1017 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:43:40.304144 1017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e940c5cdfe8c423383b2d49ff8ab2bb9: Generated new TSK 0
I20260926 20:43:40.304332 1017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260926 20:43:40.330775 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35625
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:40.410637 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:40.410857 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:40.410880 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:40.412031 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:40.412070 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:40.412096 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:43:40.413466 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35625
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:40.413641 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:40.413813 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:40.418692 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:40.418701 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:40.418828 1025 server_base.cc:1034] running on GCE node
W20260926 20:43:40.418710 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:40.419137 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:40.419378 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:40.420512 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790455420420503 us; error 26 us; skew 500 ppm
Sep 26 20:43:40 dist-test-slave-4hd3 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:40.422924 1025 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:43:40.423333 1025 webserver.cc:460] Webserver started at http://127.0.98.65:41863/ using document root <none> and password file <none>
I20260926 20:43:40.423586 1025 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:40.423674 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:40.423832 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:40.424921 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "546772fae669424999a62655fa19c9dc"
format_stamp: "Formatted at 2026-09-26 20:43:40 on dist-test-slave-4hd3"
server_key: "458bdcada198ea0720f71a19ed0a72f1"
server_key_iv: "4a9d34d520fcc65fbf5b2358e5b30e0e"
server_key_version: "encryptionkey@0"
I20260926 20:43:40.425288 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "546772fae669424999a62655fa19c9dc"
format_stamp: "Formatted at 2026-09-26 20:43:40 on dist-test-slave-4hd3"
server_key: "458bdcada198ea0720f71a19ed0a72f1"
server_key_iv: "4a9d34d520fcc65fbf5b2358e5b30e0e"
server_key_version: "encryptionkey@0"
I20260926 20:43:40.426549 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:43:40.427240 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:40.427415 1025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:43:40.427513 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "546772fae669424999a62655fa19c9dc"
format_stamp: "Formatted at 2026-09-26 20:43:40 on dist-test-slave-4hd3"
server_key: "458bdcada198ea0720f71a19ed0a72f1"
server_key_iv: "4a9d34d520fcc65fbf5b2358e5b30e0e"
server_key_version: "encryptionkey@0"
I20260926 20:43:40.427591 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:40.435532 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:40.437301 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:40.437458 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:40.437669 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:40.437970 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:40.438024 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:40.438071 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:40.438099 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:40.443683 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:34225
I20260926 20:43:40.443799 1153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:34225 every 8 connection(s)
I20260926 20:43:40.444080 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:43:40.445195 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 1025
I20260926 20:43:40.445297 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:43:40.445454 393 external_mini_cluster.cc:1428] Setting key 6fa1f6878bb2c02d0add3033c72058db
Sep 26 20:43:40 dist-test-slave-4hd3 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260926 20:43:40.454779 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:40.444807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:42651 (local address 127.0.98.126:35625)
0926 20:43:40.444846 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:40.444848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:40.445051 (+ 203us) server_negotiation.cc:407] Connection header received
0926 20:43:40.445615 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:40.445617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:40.445632 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:40.445692 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:40.446784 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.447537 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.449157 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.449437 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.449511 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:40.451947 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:40.451948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:40.451948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:40.451956 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:40.453370 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:40.453929 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:40.453930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:40.453931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:40.453981 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:40.454247 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:40.454248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:40.454248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:40.454348 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:40.454399 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:40.454636 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 20:43:40.454707 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 20:43:40.454957 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:40.444756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35625 (local address 127.0.98.65:42651)
0926 20:43:40.444914 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:40.444955 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:40.445475 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:40.445863 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:40.445867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:40.446459 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:40.446674 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:40.446679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.447627 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.447627 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:40.449036 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:40.449039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.449550 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.449551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:40.449718 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:40.450208 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:40.450218 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:40.451854 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:40.453566 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:40.453574 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:40.453576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:40.453861 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:40.454068 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:40.454069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:40.454069 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:40.454180 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:40.454407 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:40.454411 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:40.454526 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 20:43:40.454610 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:43:40.454738 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260926 20:43:40.455544 1154 heartbeater.cc:347] Connected to a master server at 127.0.98.126:35625
I20260926 20:43:40.455648 1154 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:40.455974 1154 heartbeater.cc:511] Master 127.0.98.126:35625 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260926 20:43:40.456586 963 ts_manager.cc:194] Registered new tserver with Master: 546772fae669424999a62655fa19c9dc (127.0.98.65:34225)
I20260926 20:43:40.457547 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:42651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260926 20:43:40.471390 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35625
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:40.551497 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:40.551728 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:40.551791 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:40.552930 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:40.553005 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:40.553051 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:43:40.554466 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35625
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:40.554725 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:40.554953 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:40.559805 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:40.559809 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:40.559944 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:40.560075 1159 server_base.cc:1034] running on GCE node
I20260926 20:43:40.560343 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:40.560590 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:40.561756 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790455420561732 us; error 43 us; skew 500 ppm
Sep 26 20:43:40 dist-test-slave-4hd3 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:40.564502 1159 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:43:40.565040 1159 webserver.cc:460] Webserver started at http://127.0.98.66:41393/ using document root <none> and password file <none>
I20260926 20:43:40.565279 1159 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:40.565344 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:40.565435 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:40.566411 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "08bad32214694980b51d7c9277ff5f1f"
format_stamp: "Formatted at 2026-09-26 20:43:40 on dist-test-slave-4hd3"
server_key: "b353b5abdc83a4eb4b767c8e0640348d"
server_key_iv: "d5197a49c7d051f4d0ce853045242b65"
server_key_version: "encryptionkey@0"
I20260926 20:43:40.566708 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "08bad32214694980b51d7c9277ff5f1f"
format_stamp: "Formatted at 2026-09-26 20:43:40 on dist-test-slave-4hd3"
server_key: "b353b5abdc83a4eb4b767c8e0640348d"
server_key_iv: "d5197a49c7d051f4d0ce853045242b65"
server_key_version: "encryptionkey@0"
I20260926 20:43:40.568003 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:43:40.568701 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:40.568980 1159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:43:40.569078 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "08bad32214694980b51d7c9277ff5f1f"
format_stamp: "Formatted at 2026-09-26 20:43:40 on dist-test-slave-4hd3"
server_key: "b353b5abdc83a4eb4b767c8e0640348d"
server_key_iv: "d5197a49c7d051f4d0ce853045242b65"
server_key_version: "encryptionkey@0"
I20260926 20:43:40.569150 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:40.591439 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:40.593472 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:40.593595 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:40.593875 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:40.594187 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:40.594244 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:40.594285 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:40.594312 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:40.600359 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:45233
I20260926 20:43:40.600405 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:45233 every 8 connection(s)
I20260926 20:43:40.600723 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:43:40.605960 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 1159
I20260926 20:43:40.606055 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:43:40.606241 393 external_mini_cluster.cc:1428] Setting key 99799f81f6a98ec1615c56a42c6a1ea7
Sep 26 20:43:40 dist-test-slave-4hd3 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:40.610769 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:40.601679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:35589 (local address 127.0.98.126:35625)
0926 20:43:40.601739 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:40.601741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:40.601959 (+ 218us) server_negotiation.cc:407] Connection header received
0926 20:43:40.602481 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:40.602482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:40.602494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:40.602556 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:40.603569 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.604107 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.605132 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.605323 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.605412 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:40.607801 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:40.607803 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:40.607803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:40.607812 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:40.609401 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:40.609795 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:40.609796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:40.609797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:40.609855 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:40.610144 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:40.610145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:40.610145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:40.610251 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:40.610326 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:40.610594 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 20:43:40.610670 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 20:43:40.610965 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:40.601638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35625 (local address 127.0.98.66:35589)
0926 20:43:40.601825 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:40.601871 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:40.602375 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:40.602639 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:40.602643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:40.603222 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:40.603412 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:40.603418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.604220 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.604221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:40.605029 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:40.605032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.605434 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.605435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:40.605559 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:40.606175 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:40.606183 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:40.607670 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:40.609490 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:40.609495 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:40.609496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:40.609718 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:40.609940 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:40.609941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:40.609941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:40.610058 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:40.610317 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:40.610323 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:40.610462 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 20:43:40.610571 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 20:43:40.610726 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
I20260926 20:43:40.611443 1288 heartbeater.cc:347] Connected to a master server at 127.0.98.126:35625
I20260926 20:43:40.611552 1288 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:40.611796 1288 heartbeater.cc:511] Master 127.0.98.126:35625 requested a full tablet report, sending...
I20260926 20:43:40.612221 963 ts_manager.cc:194] Registered new tserver with Master: 08bad32214694980b51d7c9277ff5f1f (127.0.98.66:45233)
I20260926 20:43:40.612766 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:35589
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260926 20:43:40.632236 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35625
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:40.712131 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:40.712321 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:40.712344 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:40.713434 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:40.713470 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:40.713501 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:43:40.714824 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35625
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:40.714993 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:40.715154 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:40.720012 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:40.720012 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:40.720041 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:40.720316 1293 server_base.cc:1034] running on GCE node
I20260926 20:43:40.720573 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:40.720878 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:40.722054 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790455420722047 us; error 36 us; skew 500 ppm
Sep 26 20:43:40 dist-test-slave-4hd3 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:40.724588 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:43:40.725066 1293 webserver.cc:460] Webserver started at http://127.0.98.67:39839/ using document root <none> and password file <none>
I20260926 20:43:40.725288 1293 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:40.725353 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:40.725507 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:40.726595 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "850e1765f79f41ff9932956f85c90228"
format_stamp: "Formatted at 2026-09-26 20:43:40 on dist-test-slave-4hd3"
server_key: "24dd0bd96475ea87d3464bc58c486aa0"
server_key_iv: "cb60cdaa01ac45a5877e4a052fd326e6"
server_key_version: "encryptionkey@0"
I20260926 20:43:40.726958 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "850e1765f79f41ff9932956f85c90228"
format_stamp: "Formatted at 2026-09-26 20:43:40 on dist-test-slave-4hd3"
server_key: "24dd0bd96475ea87d3464bc58c486aa0"
server_key_iv: "cb60cdaa01ac45a5877e4a052fd326e6"
server_key_version: "encryptionkey@0"
I20260926 20:43:40.728308 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:43:40.729030 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:40.729197 1293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:43:40.729300 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "850e1765f79f41ff9932956f85c90228"
format_stamp: "Formatted at 2026-09-26 20:43:40 on dist-test-slave-4hd3"
server_key: "24dd0bd96475ea87d3464bc58c486aa0"
server_key_iv: "cb60cdaa01ac45a5877e4a052fd326e6"
server_key_version: "encryptionkey@0"
I20260926 20:43:40.729374 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:40.754006 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:40.756108 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:40.756273 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:40.756552 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:40.756873 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:40.756907 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:40.756930 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:40.756942 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:40.762638 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:32773
I20260926 20:43:40.762704 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:32773 every 8 connection(s)
I20260926 20:43:40.763018 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:43:40.766920 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 1293
I20260926 20:43:40.767009 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:43:40.767174 393 external_mini_cluster.cc:1428] Setting key 0ef721f34e5fc0adf96c61efa662408a
Sep 26 20:43:40 dist-test-slave-4hd3 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:40.772971 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:40.763778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:38607 (local address 127.0.98.126:35625)
0926 20:43:40.763847 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:40.763849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:40.764068 (+ 219us) server_negotiation.cc:407] Connection header received
0926 20:43:40.764593 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:40.764594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:40.764606 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:40.764658 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:40.765560 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.766153 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.767396 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.767673 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.767776 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:40.769978 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:40.769980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:40.769981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:40.769988 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:40.771418 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:40.771981 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:40.771982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:40.771983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:40.772049 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:40.772311 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:40.772312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:40.772312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:40.772424 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:40.772498 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:40.772788 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 20:43:40.772866 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:43:40.773155 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:40.763785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35625 (local address 127.0.98.67:38607)
0926 20:43:40.763951 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:40.763986 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:40.764478 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:40.764734 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:40.764737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:40.765278 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:40.765464 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:40.765468 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.766244 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.766245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:40.767284 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:40.767287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.767785 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.767786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:40.767909 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:40.768442 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:40.768449 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:40.769872 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:40.771521 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:40.771528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:40.771530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:40.771888 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:40.772116 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:40.772117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:40.772117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:40.772234 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:40.772490 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:40.772496 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:40.772644 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 20:43:40.772755 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 20:43:40.772922 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260926 20:43:40.773646 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.126:35625
I20260926 20:43:40.773762 1422 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:40.773941 1422 heartbeater.cc:511] Master 127.0.98.126:35625 requested a full tablet report, sending...
I20260926 20:43:40.774313 963 ts_manager.cc:194] Registered new tserver with Master: 850e1765f79f41ff9932956f85c90228 (127.0.98.67:32773)
I20260926 20:43:40.774696 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:38607
I20260926 20:43:40.778920 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:43:40.786478 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:40.780632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56698 (local address 127.0.98.126:35625)
0926 20:43:40.780715 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:40.780717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:40.780810 (+ 93us) server_negotiation.cc:407] Connection header received
0926 20:43:40.780887 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:40.780888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:40.780898 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:40.780954 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:40.781506 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.782008 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.782729 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.782932 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.783004 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:40.784194 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:40.784195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:40.784196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:40.784203 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:40.785327 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:40.785681 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:40.785682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:40.785682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:40.785741 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:40.785928 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:40.785929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:40.785930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:40.786070 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:40.786170 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:40.786302 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 20:43:40.786379 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:43:40.788538 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:43:40.790071 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:43:40.796483 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:40.792075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34225 (local address 127.0.0.1:37872)
0926 20:43:40.792416 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:40.792430 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:40.792493 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:40.792705 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:40.792708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:40.792710 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:40.792781 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:40.792782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.793867 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.793868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:40.795097 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:40.795099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.796098 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.796099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:40.796246 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:40.796265 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 20:43:40.796340 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:43:40.796403 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":44,"thread_start_us":68,"threads_started":1}
I20260926 20:43:40.796478 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:40.792227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37872 (local address 127.0.98.65:34225)
0926 20:43:40.792406 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:40.792411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:40.792479 (+ 68us) server_negotiation.cc:407] Connection header received
0926 20:43:40.792578 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:40.792580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:40.792598 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:40.792686 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:40.792978 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.793767 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.795192 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.795994 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.796182 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:40.796239 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:40.796354 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 20:43:40.796396 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260926 20:43:40.796638 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:40.792245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.0.98.66:45233)
0926 20:43:40.792707 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:40.792712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:40.792721 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:43:40.792756 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:40.792758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:40.792773 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:40.792851 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:40.792962 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.794072 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.795286 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.795994 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.796069 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:40.796127 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:40.796524 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 20:43:40.796567 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":80,"threads_started":1}
I20260926 20:43:40.796486 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:40.792076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:45233 (local address 127.0.0.1:57112)
0926 20:43:40.792393 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:40.792429 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:40.792491 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:40.792856 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:40.792857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:40.792858 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:40.792888 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:40.792889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.794174 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.794175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:40.795211 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:40.795213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.796124 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.796124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:40.796243 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:40.796267 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:43:40.796334 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 20:43:40.796403 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":27,"thread_start_us":70,"threads_started":1}
I20260926 20:43:40.797372 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:40.792093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:32773 (local address 127.0.0.1:35192)
0926 20:43:40.792560 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:40.792567 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:40.792592 (+ 25us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:40.792814 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:40.792814 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:40.792834 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:40.792888 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:40.792889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.795208 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.795208 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:40.795995 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:40.795997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.797121 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.797122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:40.797191 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:40.797194 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:43:40.797253 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 20:43:40.797297 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":55,"threads_started":1}
I20260926 20:43:40.797282 1088 tablet_service.cc:1467] Processing CreateTablet for tablet 81d20b2c91974d889ac4e1ac0b6dc77c (DEFAULT_TABLE table=test-table [id=6ec1ef19d38047349dc5c6fabbd63207]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:40.797396 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:40.792289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35192 (local address 127.0.98.67:32773)
0926 20:43:40.792616 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:40.792621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:40.792631 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:43:40.792665 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:40.792667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:40.792681 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:40.792747 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:40.792990 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.794076 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.796079 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:40.797012 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:40.797101 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:40.797155 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:40.797286 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 20:43:40.797330 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":79,"threads_started":1}
I20260926 20:43:40.797605 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d20b2c91974d889ac4e1ac0b6dc77c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:40.797966 1356 tablet_service.cc:1467] Processing CreateTablet for tablet 81d20b2c91974d889ac4e1ac0b6dc77c (DEFAULT_TABLE table=test-table [id=6ec1ef19d38047349dc5c6fabbd63207]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:40.798206 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d20b2c91974d889ac4e1ac0b6dc77c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:40.800029 1436 tablet_bootstrap.cc:492] T 81d20b2c91974d889ac4e1ac0b6dc77c P 546772fae669424999a62655fa19c9dc: Bootstrap starting.
I20260926 20:43:40.800283 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 81d20b2c91974d889ac4e1ac0b6dc77c (DEFAULT_TABLE table=test-table [id=6ec1ef19d38047349dc5c6fabbd63207]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:40.800582 1436 tablet_bootstrap.cc:654] T 81d20b2c91974d889ac4e1ac0b6dc77c P 546772fae669424999a62655fa19c9dc: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:40.800595 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d20b2c91974d889ac4e1ac0b6dc77c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:40.800664 1437 tablet_bootstrap.cc:492] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228: Bootstrap starting.
I20260926 20:43:40.801165 1436 log.cc:824] T 81d20b2c91974d889ac4e1ac0b6dc77c P 546772fae669424999a62655fa19c9dc: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:40.801327 1437 tablet_bootstrap.cc:654] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:40.801860 1437 log.cc:824] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:40.801944 1436 tablet_bootstrap.cc:492] T 81d20b2c91974d889ac4e1ac0b6dc77c P 546772fae669424999a62655fa19c9dc: No bootstrap required, opened a new log
I20260926 20:43:40.802000 1436 ts_tablet_manager.cc:1397] T 81d20b2c91974d889ac4e1ac0b6dc77c P 546772fae669424999a62655fa19c9dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:43:40.802597 1437 tablet_bootstrap.cc:492] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228: No bootstrap required, opened a new log
I20260926 20:43:40.802650 1437 ts_tablet_manager.cc:1397] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:43:40.803195 1440 tablet_bootstrap.cc:492] T 81d20b2c91974d889ac4e1ac0b6dc77c P 08bad32214694980b51d7c9277ff5f1f: Bootstrap starting.
I20260926 20:43:40.803833 1440 tablet_bootstrap.cc:654] T 81d20b2c91974d889ac4e1ac0b6dc77c P 08bad32214694980b51d7c9277ff5f1f: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:40.804395 1436 raft_consensus.cc:348] T 81d20b2c91974d889ac4e1ac0b6dc77c P 546772fae669424999a62655fa19c9dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bad32214694980b51d7c9277ff5f1f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45233 } } peers { permanent_uuid: "546772fae669424999a62655fa19c9dc" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34225 } } peers { permanent_uuid: "850e1765f79f41ff9932956f85c90228" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 32773 } }
I20260926 20:43:40.804705 1436 raft_consensus.cc:374] T 81d20b2c91974d889ac4e1ac0b6dc77c P 546772fae669424999a62655fa19c9dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:40.804754 1436 raft_consensus.cc:729] T 81d20b2c91974d889ac4e1ac0b6dc77c P 546772fae669424999a62655fa19c9dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 546772fae669424999a62655fa19c9dc, State: Initialized, Role: FOLLOWER
I20260926 20:43:40.804859 1440 log.cc:824] T 81d20b2c91974d889ac4e1ac0b6dc77c P 08bad32214694980b51d7c9277ff5f1f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:40.804864 1436 consensus_queue.cc:260] T 81d20b2c91974d889ac4e1ac0b6dc77c P 546772fae669424999a62655fa19c9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bad32214694980b51d7c9277ff5f1f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45233 } } peers { permanent_uuid: "546772fae669424999a62655fa19c9dc" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34225 } } peers { permanent_uuid: "850e1765f79f41ff9932956f85c90228" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 32773 } }
I20260926 20:43:40.804993 1437 raft_consensus.cc:348] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bad32214694980b51d7c9277ff5f1f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45233 } } peers { permanent_uuid: "546772fae669424999a62655fa19c9dc" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34225 } } peers { permanent_uuid: "850e1765f79f41ff9932956f85c90228" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 32773 } }
I20260926 20:43:40.805127 1436 ts_tablet_manager.cc:1428] T 81d20b2c91974d889ac4e1ac0b6dc77c P 546772fae669424999a62655fa19c9dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:43:40.805154 1437 raft_consensus.cc:374] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:40.805189 1437 raft_consensus.cc:729] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 850e1765f79f41ff9932956f85c90228, State: Initialized, Role: FOLLOWER
I20260926 20:43:40.805271 1154 heartbeater.cc:503] Master 127.0.98.126:35625 was elected leader, sending a full tablet report...
I20260926 20:43:40.805336 1437 consensus_queue.cc:260] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bad32214694980b51d7c9277ff5f1f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45233 } } peers { permanent_uuid: "546772fae669424999a62655fa19c9dc" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34225 } } peers { permanent_uuid: "850e1765f79f41ff9932956f85c90228" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 32773 } }
I20260926 20:43:40.805619 1422 heartbeater.cc:503] Master 127.0.98.126:35625 was elected leader, sending a full tablet report...
I20260926 20:43:40.805668 1440 tablet_bootstrap.cc:492] T 81d20b2c91974d889ac4e1ac0b6dc77c P 08bad32214694980b51d7c9277ff5f1f: No bootstrap required, opened a new log
I20260926 20:43:40.805727 1440 ts_tablet_manager.cc:1397] T 81d20b2c91974d889ac4e1ac0b6dc77c P 08bad32214694980b51d7c9277ff5f1f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 20:43:40.805606 1437 ts_tablet_manager.cc:1428] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 20:43:40.807335 1440 raft_consensus.cc:348] T 81d20b2c91974d889ac4e1ac0b6dc77c P 08bad32214694980b51d7c9277ff5f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bad32214694980b51d7c9277ff5f1f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45233 } } peers { permanent_uuid: "546772fae669424999a62655fa19c9dc" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34225 } } peers { permanent_uuid: "850e1765f79f41ff9932956f85c90228" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 32773 } }
I20260926 20:43:40.807458 1440 raft_consensus.cc:374] T 81d20b2c91974d889ac4e1ac0b6dc77c P 08bad32214694980b51d7c9277ff5f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:40.807479 1440 raft_consensus.cc:729] T 81d20b2c91974d889ac4e1ac0b6dc77c P 08bad32214694980b51d7c9277ff5f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bad32214694980b51d7c9277ff5f1f, State: Initialized, Role: FOLLOWER
I20260926 20:43:40.807585 1440 consensus_queue.cc:260] T 81d20b2c91974d889ac4e1ac0b6dc77c P 08bad32214694980b51d7c9277ff5f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bad32214694980b51d7c9277ff5f1f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45233 } } peers { permanent_uuid: "546772fae669424999a62655fa19c9dc" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34225 } } peers { permanent_uuid: "850e1765f79f41ff9932956f85c90228" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 32773 } }
I20260926 20:43:40.807858 1440 ts_tablet_manager.cc:1428] T 81d20b2c91974d889ac4e1ac0b6dc77c P 08bad32214694980b51d7c9277ff5f1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 20:43:40.807883 1288 heartbeater.cc:503] Master 127.0.98.126:35625 was elected leader, sending a full tablet report...
W20260926 20:43:40.851579 1289 tablet.cc:2369] T 81d20b2c91974d889ac4e1ac0b6dc77c P 08bad32214694980b51d7c9277ff5f1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:43:40.944823 1155 tablet.cc:2369] T 81d20b2c91974d889ac4e1ac0b6dc77c P 546772fae669424999a62655fa19c9dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:43:41.005152 1443 raft_consensus.cc:482] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:43:41.005371 1443 raft_consensus.cc:504] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bad32214694980b51d7c9277ff5f1f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45233 } } peers { permanent_uuid: "546772fae669424999a62655fa19c9dc" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34225 } } peers { permanent_uuid: "850e1765f79f41ff9932956f85c90228" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 32773 } }
I20260926 20:43:41.005891 1443 leader_election.cc:288] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08bad32214694980b51d7c9277ff5f1f (127.0.98.66:45233), 546772fae669424999a62655fa19c9dc (127.0.98.65:34225)
I20260926 20:43:41.009114 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:41.006215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:38751 (local address 127.0.98.65:34225)
0926 20:43:41.006286 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:41.006289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:41.006448 (+ 159us) server_negotiation.cc:407] Connection header received
0926 20:43:41.006478 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:41.006479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:41.006489 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:41.006563 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:41.006739 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:41.007216 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:41.008394 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:41.008743 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:41.008804 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:41.008814 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:41.008993 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 20:43:41.009042 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:43:41.009124 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:41.006143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34225 (local address 127.0.98.67:38751)
0926 20:43:41.006358 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:41.006368 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:41.006424 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:41.006572 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:41.006573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:41.006575 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:41.006641 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:41.006643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:41.007330 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:41.007331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:41.008267 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:41.008270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:41.008829 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:41.008830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:41.008908 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:41.008912 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:43:41.008955 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 20:43:41.008991 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":37,"thread_start_us":144,"threads_started":1}
I20260926 20:43:41.009158 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:41.006283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:57109 (local address 127.0.98.66:45233)
0926 20:43:41.006332 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:41.006334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:41.006342 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:43:41.006373 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:41.006374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:41.006389 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:41.006448 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:41.006693 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:41.007197 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:41.008353 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:41.008762 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:41.008855 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:41.008866 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:41.009040 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 20:43:41.009084 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:43:41.009138 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:41.006086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:45233 (local address 127.0.98.67:57109)
0926 20:43:41.006179 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:41.006193 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:41.006250 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:41.006522 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:41.006524 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:41.006526 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:41.006604 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:41.006607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:41.007298 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:41.007299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:41.008251 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:41.008254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:41.008870 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:41.008871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:41.008957 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:41.008961 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:43:41.009022 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 20:43:41.009072 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":3}
I20260926 20:43:41.009538 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d20b2c91974d889ac4e1ac0b6dc77c" candidate_uuid: "850e1765f79f41ff9932956f85c90228" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "546772fae669424999a62655fa19c9dc" is_pre_election: true
I20260926 20:43:41.009533 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d20b2c91974d889ac4e1ac0b6dc77c" candidate_uuid: "850e1765f79f41ff9932956f85c90228" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bad32214694980b51d7c9277ff5f1f" is_pre_election: true
I20260926 20:43:41.009707 1242 raft_consensus.cc:2454] T 81d20b2c91974d889ac4e1ac0b6dc77c P 08bad32214694980b51d7c9277ff5f1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 850e1765f79f41ff9932956f85c90228 in term 0.
I20260926 20:43:41.009707 1108 raft_consensus.cc:2454] T 81d20b2c91974d889ac4e1ac0b6dc77c P 546772fae669424999a62655fa19c9dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 850e1765f79f41ff9932956f85c90228 in term 0.
I20260926 20:43:41.009967 1310 leader_election.cc:302] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bad32214694980b51d7c9277ff5f1f, 850e1765f79f41ff9932956f85c90228; no voters:
I20260926 20:43:41.010216 1443 raft_consensus.cc:2781] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:43:41.010278 1443 raft_consensus.cc:482] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:43:41.010308 1443 raft_consensus.cc:3037] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:41.011053 1443 raft_consensus.cc:504] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bad32214694980b51d7c9277ff5f1f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45233 } } peers { permanent_uuid: "546772fae669424999a62655fa19c9dc" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34225 } } peers { permanent_uuid: "850e1765f79f41ff9932956f85c90228" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 32773 } }
I20260926 20:43:41.011232 1443 leader_election.cc:288] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228 [CANDIDATE]: Term 1 election: Requested vote from peers 08bad32214694980b51d7c9277ff5f1f (127.0.98.66:45233), 546772fae669424999a62655fa19c9dc (127.0.98.65:34225)
I20260926 20:43:41.011407 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d20b2c91974d889ac4e1ac0b6dc77c" candidate_uuid: "850e1765f79f41ff9932956f85c90228" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bad32214694980b51d7c9277ff5f1f"
I20260926 20:43:41.011499 1242 raft_consensus.cc:3037] T 81d20b2c91974d889ac4e1ac0b6dc77c P 08bad32214694980b51d7c9277ff5f1f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:41.011469 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d20b2c91974d889ac4e1ac0b6dc77c" candidate_uuid: "850e1765f79f41ff9932956f85c90228" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "546772fae669424999a62655fa19c9dc"
I20260926 20:43:41.011585 1108 raft_consensus.cc:3037] T 81d20b2c91974d889ac4e1ac0b6dc77c P 546772fae669424999a62655fa19c9dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:41.012320 1242 raft_consensus.cc:2454] T 81d20b2c91974d889ac4e1ac0b6dc77c P 08bad32214694980b51d7c9277ff5f1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 850e1765f79f41ff9932956f85c90228 in term 1.
I20260926 20:43:41.012388 1108 raft_consensus.cc:2454] T 81d20b2c91974d889ac4e1ac0b6dc77c P 546772fae669424999a62655fa19c9dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 850e1765f79f41ff9932956f85c90228 in term 1.
I20260926 20:43:41.012528 1310 leader_election.cc:302] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bad32214694980b51d7c9277ff5f1f, 850e1765f79f41ff9932956f85c90228; no voters:
I20260926 20:43:41.012648 1443 raft_consensus.cc:2781] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:41.012868 1443 raft_consensus.cc:686] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228 [term 1 LEADER]: Becoming Leader. State: Replica: 850e1765f79f41ff9932956f85c90228, State: Running, Role: LEADER
I20260926 20:43:41.012996 1443 consensus_queue.cc:237] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bad32214694980b51d7c9277ff5f1f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45233 } } peers { permanent_uuid: "546772fae669424999a62655fa19c9dc" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34225 } } peers { permanent_uuid: "850e1765f79f41ff9932956f85c90228" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 32773 } }
W20260926 20:43:41.013527 1423 tablet.cc:2369] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:43:41.013859 963 catalog_manager.cc:5495] T 81d20b2c91974d889ac4e1ac0b6dc77c P 850e1765f79f41ff9932956f85c90228 reported cstate change: term changed from 0 to 1, leader changed from <none> to 850e1765f79f41ff9932956f85c90228 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "850e1765f79f41ff9932956f85c90228" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bad32214694980b51d7c9277ff5f1f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "546772fae669424999a62655fa19c9dc" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "850e1765f79f41ff9932956f85c90228" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 32773 } health_report { overall_health: HEALTHY } } }
Sep 26 20:43:41 dist-test-slave-4hd3 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260926 20:43:41.025846 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:41.018784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37886 (local address 127.0.98.65:34225)
0926 20:43:41.018853 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:41.018855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:41.018983 (+ 128us) server_negotiation.cc:407] Connection header received
0926 20:43:41.019008 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:41.019009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:41.019017 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:41.019070 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:41.019537 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:41.020093 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:41.020767 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:41.020943 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:41.020995 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:41.023295 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:41.023303 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:41.023304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:41.023310 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:41.024873 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:41.025189 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:41.025191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:41.025192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:41.025248 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:41.025404 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:41.025404 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:41.025405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:41.025521 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:41.025653 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:41.025680 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 20:43:41.025735 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 20:43:41 dist-test-slave-4hd3 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455421, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:43:41 dist-test-slave-4hd3 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455421, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260926 20:43:41.048697 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:41.041731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37894 (local address 127.0.98.65:34225)
0926 20:43:41.041815 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:41.041817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:41.041829 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:43:41.041853 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:41.041854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:41.041863 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:41.041927 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:41.042451 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:41.042929 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:41.043708 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:41.043942 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:41.043993 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:41.046185 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:41.046186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:41.046186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:41.046192 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:41.047350 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:41.047934 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:41.047935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:41.047935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:41.047987 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:41.048221 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:41.048221 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:41.048222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:41.048310 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:41.048434 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:41.048537 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 20:43:41.048596 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 20:43:41 dist-test-slave-4hd3 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:43:41 dist-test-slave-4hd3 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260926 20:43:41.069808 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:41.063203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.0.98.65:34225)
0926 20:43:41.063261 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:41.063263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:41.063302 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:43:41.063368 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:41.063369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:41.063378 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:41.063427 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:41.064022 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:41.064493 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:41.065189 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:41.065359 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:41.065410 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:41.067519 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:41.067520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:41.067520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:41.067526 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:41.068808 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:41.069091 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:41.069091 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:41.069092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:41.069158 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:41.069360 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:41.069361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:41.069361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:41.069440 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:41.069503 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:41.069667 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 20:43:41.069720 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 20:43:41.070312 1068 tablet_service.cc:2908] Rejecting scan request for tablet 81d20b2c91974d889ac4e1ac0b6dc77c: Uninitialized: clean time has not yet been initialized
I20260926 20:43:41.071152 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 1025
I20260926 20:43:41.076040 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 1159
I20260926 20:43:41.080964 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 1293
I20260926 20:43:41.085177 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 931
2026-09-26T20:43:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2081 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:43:41 dist-test-slave-4hd3 krb5kdc[1460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:43:41 dist-test-slave-4hd3 krb5kdc[1460](info): set up 2 sockets
Sep 26 20:43:41 dist-test-slave-4hd3 krb5kdc[1460](info): commencing operation
krb5kdc: starting...
W20260926 20:43:42.111418 393 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:43:42 dist-test-slave-4hd3 krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:43:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:43:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:43:42.224757 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39137
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:42.305768 1473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:42.305974 1473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:42.305996 1473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:42.307154 1473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:43:42.307194 1473 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:43:42.307209 1473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:42.307220 1473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:42.307233 1473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:43:42.307317 1473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:43:42.308826 1473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39137
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:42.309082 1473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:42.309314 1473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:42.314082 1481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:42.314059 1479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:42.314118 1478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:42.314237 1473 server_base.cc:1034] running on GCE node
I20260926 20:43:42.314451 1473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:42.314778 1473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:42.315939 1473 hybrid_clock.cc:648] HybridClock initialized: now 1790455422315917 us; error 39 us; skew 500 ppm
Sep 26 20:43:42 dist-test-slave-4hd3 krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:42.318403 1473 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:43:42.318897 1473 webserver.cc:460] Webserver started at http://127.0.98.126:33727/ using document root <none> and password file <none>
I20260926 20:43:42.319120 1473 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:42.319200 1473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:42.319353 1473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:42.320436 1473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "4a0a59e088e34c5c8aceaf186c04c2d9"
format_stamp: "Formatted at 2026-09-26 20:43:42 on dist-test-slave-4hd3"
server_key: "64a21ec2179b38a3f08b4a73682efbb7"
server_key_iv: "09e561bac101796d0ab363e1e057a996"
server_key_version: "encryptionkey@0"
I20260926 20:43:42.320801 1473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "4a0a59e088e34c5c8aceaf186c04c2d9"
format_stamp: "Formatted at 2026-09-26 20:43:42 on dist-test-slave-4hd3"
server_key: "64a21ec2179b38a3f08b4a73682efbb7"
server_key_iv: "09e561bac101796d0ab363e1e057a996"
server_key_version: "encryptionkey@0"
I20260926 20:43:42.322080 1473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:43:42.322762 1488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:42.322923 1473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:43:42.323025 1473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "4a0a59e088e34c5c8aceaf186c04c2d9"
format_stamp: "Formatted at 2026-09-26 20:43:42 on dist-test-slave-4hd3"
server_key: "64a21ec2179b38a3f08b4a73682efbb7"
server_key_iv: "09e561bac101796d0ab363e1e057a996"
server_key_version: "encryptionkey@0"
I20260926 20:43:42.323097 1473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:42.354856 1473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:42.356935 1473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:42.357120 1473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:42.361896 1473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:39137
I20260926 20:43:42.361946 1550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:39137 every 8 connection(s)
I20260926 20:43:42.362282 1473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:43:42.362918 1551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:42.365198 1551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9: Bootstrap starting.
I20260926 20:43:42.365780 1551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:42.366078 1551 log.cc:824] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:42.366742 1551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9: No bootstrap required, opened a new log
I20260926 20:43:42.368819 1551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a0a59e088e34c5c8aceaf186c04c2d9" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39137 } }
I20260926 20:43:42.368978 1551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:42.369024 1551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a0a59e088e34c5c8aceaf186c04c2d9, State: Initialized, Role: FOLLOWER
I20260926 20:43:42.369196 1551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a0a59e088e34c5c8aceaf186c04c2d9" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39137 } }
I20260926 20:43:42.369278 1551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:43:42.369319 1551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:43:42.369371 1551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:42.369472 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 1473
I20260926 20:43:42.369592 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:43:42.369765 393 external_mini_cluster.cc:1428] Setting key 4e8834e83db11289daa160594204d19d
I20260926 20:43:42.370055 1551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a0a59e088e34c5c8aceaf186c04c2d9" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39137 } }
I20260926 20:43:42.370168 1551 leader_election.cc:302] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a0a59e088e34c5c8aceaf186c04c2d9; no voters:
I20260926 20:43:42.370350 1551 leader_election.cc:288] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:43:42.370414 1556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:42.370609 1556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9 [term 1 LEADER]: Becoming Leader. State: Replica: 4a0a59e088e34c5c8aceaf186c04c2d9, State: Running, Role: LEADER
I20260926 20:43:42.370661 1551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:43:42.370776 1556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a0a59e088e34c5c8aceaf186c04c2d9" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39137 } }
I20260926 20:43:42.371407 1558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a0a59e088e34c5c8aceaf186c04c2d9. Latest consensus state: current_term: 1 leader_uuid: "4a0a59e088e34c5c8aceaf186c04c2d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a0a59e088e34c5c8aceaf186c04c2d9" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39137 } } }
I20260926 20:43:42.372008 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a0a59e088e34c5c8aceaf186c04c2d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a0a59e088e34c5c8aceaf186c04c2d9" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39137 } } }
I20260926 20:43:42.372057 1558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:42.372146 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9 [sys.catalog]: This master's current role is: LEADER
W20260926 20:43:42.372670 1571 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:43:42.372736 1571 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:43:42.372790 1572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:43:42.372948 1572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:43:42.374435 1572 catalog_manager.cc:1347] Generated new cluster ID: 1d2640b77fcc416190fe5f6fcaad05a0
I20260926 20:43:42.374491 1572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:43:42 dist-test-slave-4hd3 krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:42.380262 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.370235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54144 (local address 127.0.98.126:39137)
0926 20:43:42.370414 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:42.370424 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:42.370466 (+ 42us) server_negotiation.cc:407] Connection header received
0926 20:43:42.370597 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:42.370605 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:42.370668 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:42.370877 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:42.371795 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.373215 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.374002 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.374605 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.374713 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.377076 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:42.377079 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:42.377080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:42.377091 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:42.378743 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:42.379117 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:42.379120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:42.379120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:42.379221 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:42.379425 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:42.379426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:42.379426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:42.379655 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:42.379853 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:42.379923 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 20:43:42.380075 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260926 20:43:42.381382 1572 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:43:42.381871 1572 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:43:42.389199 1572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9: Generated new TSK 0
I20260926 20:43:42.389372 1572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260926 20:43:42.426080 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39137
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:42.521081 1577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:42.521271 1577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:42.521294 1577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:42.522423 1577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:42.522464 1577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:42.522483 1577 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:43:42.522507 1577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:43:42.524021 1577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:39137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:42.524257 1577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:42.524487 1577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:42.530025 1583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:42.530005 1585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:42.530102 1577 server_base.cc:1034] running on GCE node
W20260926 20:43:42.529992 1582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:42.530396 1577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:42.530656 1577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:42.531787 1577 hybrid_clock.cc:648] HybridClock initialized: now 1790455422531765 us; error 38 us; skew 500 ppm
Sep 26 20:43:42 dist-test-slave-4hd3 krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:42.534356 1577 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:43:42.534737 1577 webserver.cc:460] Webserver started at http://127.0.98.65:37967/ using document root <none> and password file <none>
I20260926 20:43:42.534917 1577 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:42.534956 1577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:42.535045 1577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:42.536140 1577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "771f4702d1024b5ab25fd80b7c7e2d13"
format_stamp: "Formatted at 2026-09-26 20:43:42 on dist-test-slave-4hd3"
server_key: "0e0dd942f023483e8321c3bce6e31984"
server_key_iv: "e67e6d2669364a99ab89b32461104aa2"
server_key_version: "encryptionkey@0"
I20260926 20:43:42.536439 1577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "771f4702d1024b5ab25fd80b7c7e2d13"
format_stamp: "Formatted at 2026-09-26 20:43:42 on dist-test-slave-4hd3"
server_key: "0e0dd942f023483e8321c3bce6e31984"
server_key_iv: "e67e6d2669364a99ab89b32461104aa2"
server_key_version: "encryptionkey@0"
I20260926 20:43:42.537777 1577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:43:42.538479 1592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:42.538677 1577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:43:42.538769 1577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "771f4702d1024b5ab25fd80b7c7e2d13"
format_stamp: "Formatted at 2026-09-26 20:43:42 on dist-test-slave-4hd3"
server_key: "0e0dd942f023483e8321c3bce6e31984"
server_key_iv: "e67e6d2669364a99ab89b32461104aa2"
server_key_version: "encryptionkey@0"
I20260926 20:43:42.538822 1577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:42.559309 1577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:42.561491 1577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:42.561652 1577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:42.562124 1577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:42.562161 1577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:42.562183 1577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:42.562206 1577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:42.569376 1577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:45995
I20260926 20:43:42.569458 1707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:45995 every 8 connection(s)
I20260926 20:43:42.569754 1577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
Sep 26 20:43:42 dist-test-slave-4hd3 krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:42.571414 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 1577
I20260926 20:43:42.571513 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:43:42.571764 393 external_mini_cluster.cc:1428] Setting key 2427f368da096214a90be996ccc933ae
I20260926 20:43:42.573518 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.562692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39137 (local address 127.0.98.65:39469)
0926 20:43:42.562854 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:42.562889 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:42.563395 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:42.563732 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:42.563737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:42.564163 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:42.564470 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:42.564478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.565584 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.565586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:42.567071 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:42.567075 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.567597 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.567598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:42.567784 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.568397 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:42.568409 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:42.570251 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:42.572129 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:42.572136 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:42.572138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:42.572437 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:42.572689 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:42.572689 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:42.572690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:42.572788 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:42.573067 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:42.573069 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:42.573176 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 20:43:42.573250 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:43:42.573364 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260926 20:43:42.573535 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.562684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:39469 (local address 127.0.98.126:39137)
0926 20:43:42.562725 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:42.562728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:42.562976 (+ 248us) server_negotiation.cc:407] Connection header received
0926 20:43:42.563500 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:42.563501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:42.563514 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:42.563577 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:42.564626 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.565483 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.567173 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.567508 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.567567 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.570342 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:42.570343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:42.570343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:42.570350 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:42.572031 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:42.572508 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:42.572509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:42.572509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:42.572614 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:42.572852 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:42.572853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:42.572853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:42.573004 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:42.573078 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:42.573299 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 20:43:42.573396 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:43:42.574191 1708 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39137
I20260926 20:43:42.574286 1708 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:42.574513 1708 heartbeater.cc:511] Master 127.0.98.126:39137 requested a full tablet report, sending...
I20260926 20:43:42.575070 1505 ts_manager.cc:194] Registered new tserver with Master: 771f4702d1024b5ab25fd80b7c7e2d13 (127.0.98.65:45995)
I20260926 20:43:42.576051 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:39469
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260926 20:43:42.581444 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.575204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:53705 (local address 127.0.98.126:39137)
0926 20:43:42.575238 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:42.575240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:42.575464 (+ 224us) server_negotiation.cc:407] Connection header received
0926 20:43:42.575548 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:42.575549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:42.575560 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:42.575614 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:42.575977 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.576526 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.577680 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.577858 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.577918 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.578904 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:42.578905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:42.578905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:42.578911 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:42.580329 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:42.580725 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:42.580726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:42.580727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:42.580775 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:42.580950 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:42.580951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:42.580951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:42.581035 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:42.581096 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:42.581264 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 20:43:42.581325 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:43:42.581429 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.575266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39137 (local address 127.0.98.65:53705)
0926 20:43:42.575392 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:42.575402 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:42.575477 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:42.575619 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:42.575619 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:42.575856 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:42.575902 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:42.575903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.576624 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.576625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:42.577587 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:42.577590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.577941 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.577942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:42.578021 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.578426 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:42.578430 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:42.578817 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:42.580415 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:42.580416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:42.580416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:42.580655 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:42.580847 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:42.580848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:42.580848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:42.580902 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:42.581113 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:42.581113 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:42.581162 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 20:43:42.581247 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 20:43:42.581328 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":37,"thread_start_us":83,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260926 20:43:42.598768 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39137
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:42.680737 1717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:42.680950 1717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:42.680975 1717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:42.682195 1717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:42.682242 1717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:42.682264 1717 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:43:42.682281 1717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:43:42.683845 1717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:39137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:42.684074 1717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:42.684306 1717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:42.689181 1725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:42.689163 1722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:42.689347 1717 server_base.cc:1034] running on GCE node
W20260926 20:43:42.689389 1723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:42.689581 1717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:42.689813 1717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:42.690974 1717 hybrid_clock.cc:648] HybridClock initialized: now 1790455422690961 us; error 28 us; skew 500 ppm
Sep 26 20:43:42 dist-test-slave-4hd3 krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:42.693572 1717 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:43:42.694048 1717 webserver.cc:460] Webserver started at http://127.0.98.66:32903/ using document root <none> and password file <none>
I20260926 20:43:42.694281 1717 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:42.694351 1717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:42.694509 1717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:42.695549 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "8b12c302b49841309fe0f11952573243"
format_stamp: "Formatted at 2026-09-26 20:43:42 on dist-test-slave-4hd3"
server_key: "552a5124bf2775fcdd1e7020feabed18"
server_key_iv: "9f11f86ca836ae67227fe5948556060a"
server_key_version: "encryptionkey@0"
I20260926 20:43:42.695928 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "8b12c302b49841309fe0f11952573243"
format_stamp: "Formatted at 2026-09-26 20:43:42 on dist-test-slave-4hd3"
server_key: "552a5124bf2775fcdd1e7020feabed18"
server_key_iv: "9f11f86ca836ae67227fe5948556060a"
server_key_version: "encryptionkey@0"
I20260926 20:43:42.697234 1717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:43:42.697952 1732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:42.698122 1717 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:43:42.698218 1717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "8b12c302b49841309fe0f11952573243"
format_stamp: "Formatted at 2026-09-26 20:43:42 on dist-test-slave-4hd3"
server_key: "552a5124bf2775fcdd1e7020feabed18"
server_key_iv: "9f11f86ca836ae67227fe5948556060a"
server_key_version: "encryptionkey@0"
I20260926 20:43:42.698305 1717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:42.718366 1717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:42.720336 1717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:42.720501 1717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:42.721048 1717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:42.721107 1717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:42.721155 1717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:42.721202 1717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:43:42 dist-test-slave-4hd3 krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:42.728258 1717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:41593
I20260926 20:43:42.728363 1847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:41593 every 8 connection(s)
I20260926 20:43:42.728684 1717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:43:42.731002 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.721449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:49651 (local address 127.0.98.126:39137)
0926 20:43:42.721514 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:42.721516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:42.721788 (+ 272us) server_negotiation.cc:407] Connection header received
0926 20:43:42.722319 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:42.722321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:42.722334 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:42.722385 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:42.723066 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.723749 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.724779 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.724987 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.725082 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.727931 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:42.727933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:42.727934 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:42.727942 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:42.729612 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:42.730026 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:42.730027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:42.730028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:42.730093 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:42.730346 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:42.730347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:42.730347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:42.730472 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:42.730544 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:42.730833 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 20:43:42.730912 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:43:42.731194 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.721456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39137 (local address 127.0.98.66:49651)
0926 20:43:42.721651 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:42.721690 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:42.722207 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:42.722459 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:42.722462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:42.722757 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:42.722941 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:42.722973 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.723826 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.723827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:42.724686 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:42.724688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.725082 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.725083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:42.725206 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.725782 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:42.725790 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:42.727840 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:42.729711 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:42.729717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:42.729719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:42.729944 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:42.730166 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:42.730167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:42.730167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:42.730271 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:42.730549 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:42.730554 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:42.730693 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 20:43:42.730800 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 20:43:42.730958 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":78,"threads_started":1}
I20260926 20:43:42.731738 1848 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39137
I20260926 20:43:42.731817 1848 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:42.732002 1848 heartbeater.cc:511] Master 127.0.98.126:39137 requested a full tablet report, sending...
I20260926 20:43:42.732667 1505 ts_manager.cc:194] Registered new tserver with Master: 8b12c302b49841309fe0f11952573243 (127.0.98.66:41593)
I20260926 20:43:42.733081 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:49651
I20260926 20:43:42.733551 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 1717
I20260926 20:43:42.733681 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:43:42.733872 393 external_mini_cluster.cc:1428] Setting key 7f007b0e950d5fd6f7345a0ad481c732
I20260926 20:43:42.739434 1854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.732689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39137 (local address 127.0.98.66:59199)
0926 20:43:42.733247 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:42.733259 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:42.733318 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:42.733474 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:42.733475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:42.733696 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:42.733760 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:42.733761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.734639 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.734640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:42.735216 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:42.735218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.735763 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.735764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:42.735847 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.736331 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:42.736336 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:42.736612 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:42.738402 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:42.738403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:42.738403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:42.738566 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:42.738774 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:42.738775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:42.738775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:42.738834 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:42.739191 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:42.739191 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:42.739228 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 20:43:42.739281 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 20:43:42.739358 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":74,"threads_started":1}
I20260926 20:43:42.739528 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.732762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:59199 (local address 127.0.98.126:39137)
0926 20:43:42.732794 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:42.732795 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:42.733317 (+ 522us) server_negotiation.cc:407] Connection header received
0926 20:43:42.733377 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:42.733378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:42.733388 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:42.733434 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:42.733853 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.734539 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.735296 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.735632 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.735704 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.736699 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:42.736700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:42.736700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:42.736707 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:42.738312 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:42.738637 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:42.738638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:42.738638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:42.738703 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:42.738899 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:42.738900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:42.738901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:42.739119 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:42.739197 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:42.739312 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 20:43:42.739395 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260926 20:43:42.758287 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39137
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:42.838088 1857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:42.838284 1857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:42.838306 1857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:42.839910 1857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:42.839952 1857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:42.839973 1857 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:43:42.840010 1857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:43:42.841617 1857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:39137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:42.841850 1857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:42.842062 1857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:42.847090 1865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:42.847092 1863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:42.847064 1862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:42.847368 1857 server_base.cc:1034] running on GCE node
I20260926 20:43:42.847555 1857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:42.847817 1857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:42.848979 1857 hybrid_clock.cc:648] HybridClock initialized: now 1790455422848969 us; error 26 us; skew 500 ppm
Sep 26 20:43:42 dist-test-slave-4hd3 krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:42.851497 1857 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:43:42.852025 1857 webserver.cc:460] Webserver started at http://127.0.98.67:33471/ using document root <none> and password file <none>
I20260926 20:43:42.852267 1857 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:42.852313 1857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:42.852442 1857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:42.853466 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "c5c007236ad94556b2bb497f0735b458"
format_stamp: "Formatted at 2026-09-26 20:43:42 on dist-test-slave-4hd3"
server_key: "86818d95eb1b9777202bdc3c2845b9a8"
server_key_iv: "b4a062f5e1d627d92daed548ecec3592"
server_key_version: "encryptionkey@0"
I20260926 20:43:42.853811 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "c5c007236ad94556b2bb497f0735b458"
format_stamp: "Formatted at 2026-09-26 20:43:42 on dist-test-slave-4hd3"
server_key: "86818d95eb1b9777202bdc3c2845b9a8"
server_key_iv: "b4a062f5e1d627d92daed548ecec3592"
server_key_version: "encryptionkey@0"
I20260926 20:43:42.855124 1857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:43:42.855899 1872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:42.856073 1857 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:43:42.856170 1857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "c5c007236ad94556b2bb497f0735b458"
format_stamp: "Formatted at 2026-09-26 20:43:42 on dist-test-slave-4hd3"
server_key: "86818d95eb1b9777202bdc3c2845b9a8"
server_key_iv: "b4a062f5e1d627d92daed548ecec3592"
server_key_version: "encryptionkey@0"
I20260926 20:43:42.856253 1857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:42.872363 1857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:42.874289 1857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:42.874475 1857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:42.875033 1857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:42.875072 1857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:42.875095 1857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:42.875108 1857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:43:42 dist-test-slave-4hd3 krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:42.882170 1857 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:34343
I20260926 20:43:42.882319 1987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:34343 every 8 connection(s)
I20260926 20:43:42.882818 1857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:43:42.884744 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.875438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:56945 (local address 127.0.98.126:39137)
0926 20:43:42.875512 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:42.875515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:42.875801 (+ 286us) server_negotiation.cc:407] Connection header received
0926 20:43:42.876310 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:42.876312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:42.876326 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:42.876408 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:42.877025 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.877526 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.878538 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.878732 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.878822 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.881970 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:42.881972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:42.881972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:42.882009 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:42.883534 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:42.883992 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:42.883993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:42.883993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:42.884058 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:42.884250 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:42.884251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:42.884251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:42.884348 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:42.884394 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:42.884621 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 20:43:42.884681 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:43:42.884922 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.875426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39137 (local address 127.0.98.67:56945)
0926 20:43:42.875649 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:42.875688 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:42.876217 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:42.876505 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:42.876510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:42.876751 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:42.876930 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:42.876934 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.877624 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.877625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:42.878428 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:42.878430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.878844 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.878845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:42.878964 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.879455 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:42.879463 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:42.881852 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:42.883636 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:42.883642 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:42.883643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:42.883925 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:42.884102 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:42.884102 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:42.884103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:42.884192 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:42.884402 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:42.884406 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:42.884510 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 20:43:42.884582 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 20:43:42.884724 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":154,"threads_started":1}
I20260926 20:43:42.885380 1988 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39137
I20260926 20:43:42.885459 1988 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:42.885659 1988 heartbeater.cc:511] Master 127.0.98.126:39137 requested a full tablet report, sending...
I20260926 20:43:42.886039 1504 ts_manager.cc:194] Registered new tserver with Master: c5c007236ad94556b2bb497f0735b458 (127.0.98.67:34343)
I20260926 20:43:42.886497 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:56945
I20260926 20:43:42.891918 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.886466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:51453 (local address 127.0.98.126:39137)
0926 20:43:42.886499 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:42.886500 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:42.886609 (+ 109us) server_negotiation.cc:407] Connection header received
0926 20:43:42.886630 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:42.886631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:42.886640 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:42.886703 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:42.887012 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.887470 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.888246 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.888435 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.888516 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.889402 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:42.889403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:42.889404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:42.889410 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:42.890809 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:42.891171 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:42.891172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:42.891172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:42.891230 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:42.891431 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:42.891432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:42.891432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:42.891534 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:42.891615 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:42.891763 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 20:43:42.891833 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 20:43:42.891933 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.886398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39137 (local address 127.0.98.67:51453)
0926 20:43:42.886512 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:42.886523 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:42.886575 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:42.886719 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:42.886719 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:42.886880 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:42.886934 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:42.886935 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.887560 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.887562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:42.888151 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:42.888153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.888519 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.888520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:42.888588 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.889017 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:42.889022 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:42.889296 (+ 274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:42.890919 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:42.890920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:42.890921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:42.891087 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:42.891294 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:42.891295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:42.891296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:42.891350 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:42.891599 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:42.891600 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:42.891643 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 20:43:42.891737 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:43:42.891833 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260926 20:43:42.892989 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 1857
I20260926 20:43:42.893082 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:43:42.893265 393 external_mini_cluster.cc:1428] Setting key acaba7bfc131bd5d0a01f616026f9382
I20260926 20:43:42.894093 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:43:42 dist-test-slave-4hd3 krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455422, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:43:42 dist-test-slave-4hd3 krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455422, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:42.926689 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.918024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.0.98.126:39137)
0926 20:43:42.918099 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:42.918101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:42.918111 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:43:42.918145 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:42.918146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:42.918161 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:42.918231 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:42.919164 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.919984 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.921254 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.921437 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.921488 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.924146 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:42.924147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:42.924147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:42.924154 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:42.925601 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:42.925976 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:42.925977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:42.925977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:42.926025 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:42.926283 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:42.926284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:42.926284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:42.926384 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:42.926459 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:42.926549 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 20:43:42.926601 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 20:43:42.928414 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:43:42.929972 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:43:42.936446 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.932453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.0.98.67:34343)
0926 20:43:42.932872 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:42.932879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:42.932890 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:43:42.932931 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:42.932934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:42.932950 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:42.933232 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:42.933290 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.934018 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.935360 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.936041 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.936139 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.936219 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:42.936319 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 20:43:42.936355 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":84,"threads_started":1}
I20260926 20:43:42.936439 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.932372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:34343 (local address 127.0.0.1:46804)
0926 20:43:42.932525 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:42.932577 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:42.932638 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:42.933064 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:42.933068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:42.933070 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:42.933153 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:42.933155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.934102 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.934103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:42.935277 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:42.935280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.936140 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.936141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:42.936216 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.936230 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 20:43:42.936304 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:43:42.936354 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":49,"threads_started":1}
I20260926 20:43:42.936877 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.932402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:41593 (local address 127.0.0.1:35274)
0926 20:43:42.932850 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:42.932861 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:42.932927 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:42.933205 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:42.933206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:42.933208 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:42.933262 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:42.933264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.934543 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.934544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:42.935646 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:42.935649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.936619 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.936620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:42.936672 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.936681 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 20:43:42.936739 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:43:42.936784 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":29,"thread_start_us":81,"threads_started":1}
I20260926 20:43:42.936878 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.932402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:45995 (local address 127.0.0.1:52570)
0926 20:43:42.933022 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:42.933031 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:42.933071 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:42.933306 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:42.933307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:42.933309 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:42.933361 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:42.933362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.934591 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.934592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:42.935585 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:42.935586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.936586 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.936588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:42.936672 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.936679 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:43:42.936738 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 20:43:42.936784 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":9,"spinlock_wait_cycles":768,"thread_start_us":92,"threads_started":1}
I20260926 20:43:42.937224 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.932657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.0.98.65:45995)
0926 20:43:42.932817 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:42.932822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:42.933123 (+ 301us) server_negotiation.cc:407] Connection header received
0926 20:43:42.933164 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:42.933166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:42.933181 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:42.933246 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:42.933434 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.934505 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.935676 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.936403 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.936477 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.936549 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:42.937082 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 20:43:42.937125 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260926 20:43:42.937397 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:42.932615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35274 (local address 127.0.98.66:41593)
0926 20:43:42.932823 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:42.932827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:42.932920 (+ 93us) server_negotiation.cc:407] Connection header received
0926 20:43:42.933035 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:42.933043 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:42.933061 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:42.933136 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:42.933346 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.934449 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.935773 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:42.936468 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:42.936526 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:42.936592 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:42.936767 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 20:43:42.936809 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":86,"threads_started":1}
I20260926 20:43:42.937496 1922 tablet_service.cc:1467] Processing CreateTablet for tablet 0443bb8baa1d44bfa18aaa242544195b (DEFAULT_TABLE table=test-table [id=c1c96efa228b4239ac06a91aff573d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:42.938011 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0443bb8baa1d44bfa18aaa242544195b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:42.938063 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 0443bb8baa1d44bfa18aaa242544195b (DEFAULT_TABLE table=test-table [id=c1c96efa228b4239ac06a91aff573d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:42.938177 1782 tablet_service.cc:1467] Processing CreateTablet for tablet 0443bb8baa1d44bfa18aaa242544195b (DEFAULT_TABLE table=test-table [id=c1c96efa228b4239ac06a91aff573d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:42.938390 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0443bb8baa1d44bfa18aaa242544195b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:42.938460 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0443bb8baa1d44bfa18aaa242544195b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:42.940941 2011 tablet_bootstrap.cc:492] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458: Bootstrap starting.
I20260926 20:43:42.941733 2011 tablet_bootstrap.cc:654] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:42.942128 2011 log.cc:824] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:42.942301 2012 tablet_bootstrap.cc:492] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243: Bootstrap starting.
I20260926 20:43:42.942752 2013 tablet_bootstrap.cc:492] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13: Bootstrap starting.
I20260926 20:43:42.942929 2012 tablet_bootstrap.cc:654] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:42.943001 2011 tablet_bootstrap.cc:492] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458: No bootstrap required, opened a new log
I20260926 20:43:42.943090 2011 ts_tablet_manager.cc:1397] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:43:42.943256 2012 log.cc:824] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:42.943367 2013 tablet_bootstrap.cc:654] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:42.943804 2013 log.cc:824] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:42.944172 2012 tablet_bootstrap.cc:492] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243: No bootstrap required, opened a new log
I20260926 20:43:42.944259 2012 ts_tablet_manager.cc:1397] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:43:42.944597 2013 tablet_bootstrap.cc:492] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13: No bootstrap required, opened a new log
I20260926 20:43:42.944669 2013 ts_tablet_manager.cc:1397] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:43:42.945066 2011 raft_consensus.cc:348] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } } peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } }
I20260926 20:43:42.945225 2011 raft_consensus.cc:374] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:42.945284 2011 raft_consensus.cc:729] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5c007236ad94556b2bb497f0735b458, State: Initialized, Role: FOLLOWER
I20260926 20:43:42.945406 2011 consensus_queue.cc:260] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } } peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } }
I20260926 20:43:42.945679 2011 ts_tablet_manager.cc:1428] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:43:42.945684 1988 heartbeater.cc:503] Master 127.0.98.126:39137 was elected leader, sending a full tablet report...
I20260926 20:43:42.946285 2012 raft_consensus.cc:348] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } } peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } }
I20260926 20:43:42.946456 2012 raft_consensus.cc:374] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:42.946556 2012 raft_consensus.cc:729] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b12c302b49841309fe0f11952573243, State: Initialized, Role: FOLLOWER
I20260926 20:43:42.946744 2012 consensus_queue.cc:260] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } } peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } }
I20260926 20:43:42.946679 2013 raft_consensus.cc:348] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } } peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } }
I20260926 20:43:42.946821 2013 raft_consensus.cc:374] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:42.946882 2013 raft_consensus.cc:729] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771f4702d1024b5ab25fd80b7c7e2d13, State: Initialized, Role: FOLLOWER
I20260926 20:43:42.947062 1848 heartbeater.cc:503] Master 127.0.98.126:39137 was elected leader, sending a full tablet report...
I20260926 20:43:42.947027 2013 consensus_queue.cc:260] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } } peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } }
I20260926 20:43:42.947288 2013 ts_tablet_manager.cc:1428] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:43:42.947316 1708 heartbeater.cc:503] Master 127.0.98.126:39137 was elected leader, sending a full tablet report...
I20260926 20:43:42.947543 2012 ts_tablet_manager.cc:1428] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260926 20:43:42.979259 1849 tablet.cc:2369] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:43:43.023581 2019 raft_consensus.cc:482] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:43:43.023702 2019 raft_consensus.cc:504] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } } peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } }
I20260926 20:43:43.024086 2019 leader_election.cc:288] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5c007236ad94556b2bb497f0735b458 (127.0.98.67:34343), 8b12c302b49841309fe0f11952573243 (127.0.98.66:41593)
I20260926 20:43:43.027273 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.024413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:37131 (local address 127.0.98.66:41593)
0926 20:43:43.024459 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:43.024462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:43.024470 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:43:43.024497 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:43.024498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:43.024509 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:43.024610 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:43.024848 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.025428 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.026463 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.026893 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.026947 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.026955 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:43.027157 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 20:43:43.027204 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 20:43:43.027275 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.024306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:41593 (local address 127.0.98.65:37131)
0926 20:43:43.024367 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:43.024382 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:43.024425 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:43.024639 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:43.024640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:43.024642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:43.024731 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.024733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.025500 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.025501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.026379 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.026382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.026966 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.026967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.027080 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.027083 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:43:43.027134 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 20:43:43.027173 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 20:43:43.027336 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.024451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:45635 (local address 127.0.98.67:34343)
0926 20:43:43.024494 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:43.024497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:43.024741 (+ 244us) server_negotiation.cc:407] Connection header received
0926 20:43:43.024769 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:43.024770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:43.024783 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:43.024848 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:43.024999 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.025514 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.026566 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.026977 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.027048 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.027057 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:43.027220 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 20:43:43.027262 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:43:43.027388 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.024306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:34343 (local address 127.0.98.65:45635)
0926 20:43:43.024662 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:43.024670 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:43.024723 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:43.024858 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:43.024858 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:43.024873 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:43.024933 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.024935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.025633 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.025634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.026484 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.026486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.027064 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.027065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.027139 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.027144 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:43:43.027200 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 20:43:43.027234 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":172,"thread_start_us":116,"threads_started":1}
I20260926 20:43:43.027706 1937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0443bb8baa1d44bfa18aaa242544195b" candidate_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c007236ad94556b2bb497f0735b458" is_pre_election: true
I20260926 20:43:43.027885 1937 raft_consensus.cc:2454] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 771f4702d1024b5ab25fd80b7c7e2d13 in term 0.
I20260926 20:43:43.027868 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0443bb8baa1d44bfa18aaa242544195b" candidate_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b12c302b49841309fe0f11952573243" is_pre_election: true
I20260926 20:43:43.028020 1802 raft_consensus.cc:2454] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 771f4702d1024b5ab25fd80b7c7e2d13 in term 0.
I20260926 20:43:43.028141 1595 leader_election.cc:302] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 771f4702d1024b5ab25fd80b7c7e2d13, c5c007236ad94556b2bb497f0735b458; no voters:
I20260926 20:43:43.028249 2019 raft_consensus.cc:2781] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:43:43.028311 2019 raft_consensus.cc:482] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:43:43.028343 2019 raft_consensus.cc:3037] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:43.029247 2019 raft_consensus.cc:504] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } } peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } }
I20260926 20:43:43.029453 2019 leader_election.cc:288] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [CANDIDATE]: Term 1 election: Requested vote from peers c5c007236ad94556b2bb497f0735b458 (127.0.98.67:34343), 8b12c302b49841309fe0f11952573243 (127.0.98.66:41593)
I20260926 20:43:43.029546 1937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0443bb8baa1d44bfa18aaa242544195b" candidate_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c007236ad94556b2bb497f0735b458"
I20260926 20:43:43.029522 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0443bb8baa1d44bfa18aaa242544195b" candidate_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b12c302b49841309fe0f11952573243"
I20260926 20:43:43.029640 1802 raft_consensus.cc:3037] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:43.029640 1937 raft_consensus.cc:3037] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:43.030524 1937 raft_consensus.cc:2454] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 771f4702d1024b5ab25fd80b7c7e2d13 in term 1.
I20260926 20:43:43.030524 1802 raft_consensus.cc:2454] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 771f4702d1024b5ab25fd80b7c7e2d13 in term 1.
I20260926 20:43:43.030826 1595 leader_election.cc:302] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 771f4702d1024b5ab25fd80b7c7e2d13, c5c007236ad94556b2bb497f0735b458; no voters:
I20260926 20:43:43.030927 2019 raft_consensus.cc:2781] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:43.031111 2019 raft_consensus.cc:686] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [term 1 LEADER]: Becoming Leader. State: Replica: 771f4702d1024b5ab25fd80b7c7e2d13, State: Running, Role: LEADER
I20260926 20:43:43.031219 2019 consensus_queue.cc:237] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } } peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } }
I20260926 20:43:43.032081 1503 catalog_manager.cc:5495] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 771f4702d1024b5ab25fd80b7c7e2d13 (127.0.98.65). New cstate: current_term: 1 leader_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } health_report { overall_health: UNKNOWN } } }
I20260926 20:43:43.053237 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.050795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52586 (local address 127.0.98.65:45995)
0926 20:43:43.050868 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:43.050869 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:43.050897 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:43:43.050926 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:43.050927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:43.050944 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:43.050994 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:43:43.051244 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.051757 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.052565 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.052755 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.052807 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.052972 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:43:43.053059 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:43:43.053116 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:43.053165 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 20:43:43.053182 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:43:43.055403 1937 raft_consensus.cc:1261] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458 [term 1 FOLLOWER]: Refusing update from remote peer 771f4702d1024b5ab25fd80b7c7e2d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:43.055403 1802 raft_consensus.cc:1261] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243 [term 1 FOLLOWER]: Refusing update from remote peer 771f4702d1024b5ab25fd80b7c7e2d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:43.055774 2019 consensus_queue.cc:1035] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:43:43.055907 2021 consensus_queue.cc:1035] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:43:43.058295 2024 mvcc.cc:205] Tried to move back new op lower bound from 7333705412832079872 to 7333705412736380928. Current Snapshot: MvccSnapshot[applied={T|T < 7333705412832079872}]
I20260926 20:43:43.058508 2025 mvcc.cc:205] Tried to move back new op lower bound from 7333705412832079872 to 7333705412736380928. Current Snapshot: MvccSnapshot[applied={T|T < 7333705412832079872}]
I20260926 20:43:43.058624 2026 mvcc.cc:205] Tried to move back new op lower bound from 7333705412832079872 to 7333705412736380928. Current Snapshot: MvccSnapshot[applied={T|T < 7333705412832079872}]
I20260926 20:43:43.062641 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:43:43.062731 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:43:43.063490 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9: Sending DeleteTablet for 3 replicas of tablet 0443bb8baa1d44bfa18aaa242544195b
I20260926 20:43:43.063862 1922 tablet_service.cc:1514] Processing DeleteTablet for tablet 0443bb8baa1d44bfa18aaa242544195b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:43 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:46804
I20260926 20:43:43.063870 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet 0443bb8baa1d44bfa18aaa242544195b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:43 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:52570
I20260926 20:43:43.063899 1782 tablet_service.cc:1514] Processing DeleteTablet for tablet 0443bb8baa1d44bfa18aaa242544195b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:43 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:35274
I20260926 20:43:43.064178 2036 tablet_replica.cc:330] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458: stopping tablet replica
I20260926 20:43:43.064409 2036 raft_consensus.cc:2229] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:43:43.064412 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:43:43.064559 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:43:43.064774 2036 raft_consensus.cc:2258] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:43.065042 2034 tablet_replica.cc:330] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13: stopping tablet replica
I20260926 20:43:43.065138 2034 raft_consensus.cc:2229] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:43:43.065270 2034 raft_consensus.cc:2258] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:43.065766 2036 ts_tablet_manager.cc:1905] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:43:43.065934 1922 tablet_service.cc:1467] Processing CreateTablet for tablet c5ac546d4a564afb9c541a113afc177f (DEFAULT_TABLE table=test-table [id=df0712d237854e09874a9e908d8583ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:43.065948 1642 tablet_service.cc:1467] Processing CreateTablet for tablet c5ac546d4a564afb9c541a113afc177f (DEFAULT_TABLE table=test-table [id=df0712d237854e09874a9e908d8583ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:43.066030 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ac546d4a564afb9c541a113afc177f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:43.066058 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ac546d4a564afb9c541a113afc177f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:43.066931 2013 tablet_bootstrap.cc:492] T c5ac546d4a564afb9c541a113afc177f P 771f4702d1024b5ab25fd80b7c7e2d13: Bootstrap starting.
I20260926 20:43:43.067361 2013 tablet_bootstrap.cc:654] T c5ac546d4a564afb9c541a113afc177f P 771f4702d1024b5ab25fd80b7c7e2d13: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:43.064930 2035 tablet_replica.cc:330] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243: stopping tablet replica
I20260926 20:43:43.067564 1782 tablet_service.cc:1467] Processing CreateTablet for tablet c5ac546d4a564afb9c541a113afc177f (DEFAULT_TABLE table=test-table [id=df0712d237854e09874a9e908d8583ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:43.067668 2011 tablet_bootstrap.cc:492] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458: Bootstrap starting.
I20260926 20:43:43.067665 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ac546d4a564afb9c541a113afc177f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:43.067664 2035 raft_consensus.cc:2229] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:43:43.067840 2035 raft_consensus.cc:2258] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:43.068081 2011 tablet_bootstrap.cc:654] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:43.068270 2035 ts_tablet_manager.cc:1905] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:43:43.069083 2012 tablet_bootstrap.cc:492] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243: Bootstrap starting.
I20260926 20:43:43.069473 2012 tablet_bootstrap.cc:654] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:43.069554 2036 ts_tablet_manager.cc:1918] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:43:43.069625 2036 log.cc:1196] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/0443bb8baa1d44bfa18aaa242544195b
I20260926 20:43:43.069700 2035 ts_tablet_manager.cc:1918] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:43:43.069761 2035 log.cc:1196] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/0443bb8baa1d44bfa18aaa242544195b
I20260926 20:43:43.069906 2036 ts_tablet_manager.cc:1939] T 0443bb8baa1d44bfa18aaa242544195b P c5c007236ad94556b2bb497f0735b458: Deleting consensus metadata
I20260926 20:43:43.070007 2035 ts_tablet_manager.cc:1939] T 0443bb8baa1d44bfa18aaa242544195b P 8b12c302b49841309fe0f11952573243: Deleting consensus metadata
I20260926 20:43:43.070262 2012 tablet_bootstrap.cc:492] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243: No bootstrap required, opened a new log
I20260926 20:43:43.070307 2012 ts_tablet_manager.cc:1397] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:43:43.070349 1491 catalog_manager.cc:4841] TS c5c007236ad94556b2bb497f0735b458 (127.0.98.67:34343): tablet 0443bb8baa1d44bfa18aaa242544195b (table test-table [id=c1c96efa228b4239ac06a91aff573d46]) successfully deleted
I20260926 20:43:43.070487 1492 catalog_manager.cc:4841] TS 8b12c302b49841309fe0f11952573243 (127.0.98.66:41593): tablet 0443bb8baa1d44bfa18aaa242544195b (table test-table [id=c1c96efa228b4239ac06a91aff573d46]) successfully deleted
I20260926 20:43:43.070466 2012 raft_consensus.cc:348] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } }
I20260926 20:43:43.070533 2012 raft_consensus.cc:374] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:43.070554 2012 raft_consensus.cc:729] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b12c302b49841309fe0f11952573243, State: Initialized, Role: FOLLOWER
I20260926 20:43:43.070611 2012 consensus_queue.cc:260] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } }
I20260926 20:43:43.070695 2012 ts_tablet_manager.cc:1428] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:43.070928 2013 tablet_bootstrap.cc:492] T c5ac546d4a564afb9c541a113afc177f P 771f4702d1024b5ab25fd80b7c7e2d13: No bootstrap required, opened a new log
I20260926 20:43:43.070972 2013 ts_tablet_manager.cc:1397] T c5ac546d4a564afb9c541a113afc177f P 771f4702d1024b5ab25fd80b7c7e2d13: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 20:43:43.071128 2013 raft_consensus.cc:348] T c5ac546d4a564afb9c541a113afc177f P 771f4702d1024b5ab25fd80b7c7e2d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } }
I20260926 20:43:43.071208 2013 raft_consensus.cc:374] T c5ac546d4a564afb9c541a113afc177f P 771f4702d1024b5ab25fd80b7c7e2d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:43.071231 2013 raft_consensus.cc:729] T c5ac546d4a564afb9c541a113afc177f P 771f4702d1024b5ab25fd80b7c7e2d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771f4702d1024b5ab25fd80b7c7e2d13, State: Initialized, Role: FOLLOWER
I20260926 20:43:43.071276 2013 consensus_queue.cc:260] T c5ac546d4a564afb9c541a113afc177f P 771f4702d1024b5ab25fd80b7c7e2d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } }
I20260926 20:43:43.071357 2013 ts_tablet_manager.cc:1428] T c5ac546d4a564afb9c541a113afc177f P 771f4702d1024b5ab25fd80b7c7e2d13: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:43.071168 2034 ts_tablet_manager.cc:1905] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:43:43.073235 2034 ts_tablet_manager.cc:1918] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:43:43.073312 2034 log.cc:1196] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-0/wal/wals/0443bb8baa1d44bfa18aaa242544195b
I20260926 20:43:43.073546 2034 ts_tablet_manager.cc:1939] T 0443bb8baa1d44bfa18aaa242544195b P 771f4702d1024b5ab25fd80b7c7e2d13: Deleting consensus metadata
I20260926 20:43:43.073971 1489 catalog_manager.cc:4841] TS 771f4702d1024b5ab25fd80b7c7e2d13 (127.0.98.65:45995): tablet 0443bb8baa1d44bfa18aaa242544195b (table test-table [id=c1c96efa228b4239ac06a91aff573d46]) successfully deleted
I20260926 20:43:43.081944 2011 tablet_bootstrap.cc:492] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458: No bootstrap required, opened a new log
I20260926 20:43:43.082021 2011 ts_tablet_manager.cc:1397] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20260926 20:43:43.082175 2011 raft_consensus.cc:348] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } }
I20260926 20:43:43.082271 2011 raft_consensus.cc:374] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:43.082306 2011 raft_consensus.cc:729] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5c007236ad94556b2bb497f0735b458, State: Initialized, Role: FOLLOWER
I20260926 20:43:43.082361 2011 consensus_queue.cc:260] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } }
I20260926 20:43:43.082455 2011 ts_tablet_manager.cc:1428] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260926 20:43:43.133335 1989 tablet.cc:2369] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:43:43.143069 2018 raft_consensus.cc:482] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:43:43.143189 2018 raft_consensus.cc:504] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } }
I20260926 20:43:43.143539 2018 leader_election.cc:288] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 771f4702d1024b5ab25fd80b7c7e2d13 (127.0.98.65:45995), c5c007236ad94556b2bb497f0735b458 (127.0.98.67:34343)
I20260926 20:43:43.146945 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.143794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:43889 (local address 127.0.98.65:45995)
0926 20:43:43.143838 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:43.143840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:43.143886 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:43:43.143919 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:43.143920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:43.143929 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:43.143980 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:43.144193 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.144911 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.146236 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.146598 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.146683 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.146693 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:43.146841 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 20:43:43.146876 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:43:43.146946 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.143708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:45995 (local address 127.0.98.66:43889)
0926 20:43:43.143775 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:43.143787 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:43.143829 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:43.144035 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:43.144037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:43.144039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:43.144109 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.144113 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.144999 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.145000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.146115 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.146118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.146684 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.146685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.146766 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.146769 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:43:43.146836 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 20:43:43.146876 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":3}
I20260926 20:43:43.147176 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.143906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:48403 (local address 127.0.98.67:34343)
0926 20:43:43.143953 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:43.143955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:43.144015 (+ 60us) server_negotiation.cc:407] Connection header received
0926 20:43:43.144086 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:43.144087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:43.144096 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:43.144149 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:43.144328 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.144785 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.146226 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.146771 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.146845 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.146855 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:43.147071 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 20:43:43.147109 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:43:43.147264 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ac546d4a564afb9c541a113afc177f" candidate_uuid: "8b12c302b49841309fe0f11952573243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" is_pre_election: true
I20260926 20:43:43.147383 1662 raft_consensus.cc:2454] T c5ac546d4a564afb9c541a113afc177f P 771f4702d1024b5ab25fd80b7c7e2d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b12c302b49841309fe0f11952573243 in term 0.
I20260926 20:43:43.147450 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.143728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:34343 (local address 127.0.98.66:48403)
0926 20:43:43.143964 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:43.143974 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:43.144025 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:43.144163 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:43.144164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:43.144165 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:43.144256 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.144258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.144867 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.144868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.146117 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.146118 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.146907 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.146908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.146979 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.146983 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:43:43.147322 (+ 339us) client_negotiation.cc:240] Negotiation successful
0926 20:43:43.147367 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":3,"thread_start_us":68,"threads_started":1}
I20260926 20:43:43.147575 1733 leader_election.cc:302] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 771f4702d1024b5ab25fd80b7c7e2d13, 8b12c302b49841309fe0f11952573243; no voters:
I20260926 20:43:43.147749 2018 raft_consensus.cc:2781] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:43:43.147737 1937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ac546d4a564afb9c541a113afc177f" candidate_uuid: "8b12c302b49841309fe0f11952573243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c007236ad94556b2bb497f0735b458" is_pre_election: true
I20260926 20:43:43.147801 2018 raft_consensus.cc:482] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:43:43.147820 1937 raft_consensus.cc:2454] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b12c302b49841309fe0f11952573243 in term 0.
I20260926 20:43:43.147831 2018 raft_consensus.cc:3037] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:43.148752 2018 raft_consensus.cc:504] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } }
I20260926 20:43:43.148904 2018 leader_election.cc:288] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [CANDIDATE]: Term 1 election: Requested vote from peers 771f4702d1024b5ab25fd80b7c7e2d13 (127.0.98.65:45995), c5c007236ad94556b2bb497f0735b458 (127.0.98.67:34343)
I20260926 20:43:43.149050 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ac546d4a564afb9c541a113afc177f" candidate_uuid: "8b12c302b49841309fe0f11952573243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771f4702d1024b5ab25fd80b7c7e2d13"
I20260926 20:43:43.149058 1937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ac546d4a564afb9c541a113afc177f" candidate_uuid: "8b12c302b49841309fe0f11952573243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c007236ad94556b2bb497f0735b458"
I20260926 20:43:43.149128 1662 raft_consensus.cc:3037] T c5ac546d4a564afb9c541a113afc177f P 771f4702d1024b5ab25fd80b7c7e2d13 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:43.149128 1937 raft_consensus.cc:3037] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:43.149902 1937 raft_consensus.cc:2454] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b12c302b49841309fe0f11952573243 in term 1.
I20260926 20:43:43.149902 1662 raft_consensus.cc:2454] T c5ac546d4a564afb9c541a113afc177f P 771f4702d1024b5ab25fd80b7c7e2d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b12c302b49841309fe0f11952573243 in term 1.
I20260926 20:43:43.150075 1733 leader_election.cc:302] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 771f4702d1024b5ab25fd80b7c7e2d13, 8b12c302b49841309fe0f11952573243; no voters:
I20260926 20:43:43.150164 2018 raft_consensus.cc:2781] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:43.150297 2018 raft_consensus.cc:686] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [term 1 LEADER]: Becoming Leader. State: Replica: 8b12c302b49841309fe0f11952573243, State: Running, Role: LEADER
I20260926 20:43:43.150367 2018 consensus_queue.cc:237] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } }
I20260926 20:43:43.151086 1503 catalog_manager.cc:5495] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b12c302b49841309fe0f11952573243 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "8b12c302b49841309fe0f11952573243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } health_report { overall_health: UNKNOWN } } }
Sep 26 20:43:43 dist-test-slave-4hd3 krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:43.195096 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.188398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39137 (local address 127.0.0.1:54166)
0926 20:43:43.188558 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:43.188569 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:43.188668 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:43.188813 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:43.188814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:43.189147 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:43.189189 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.189192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.189891 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.189892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.190414 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.190416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.190798 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.190799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.190857 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.191355 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:43.191360 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:43.192803 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:43.194189 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:43.194195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:43.194197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:43.194384 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:43.194561 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:43.194562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:43.194562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:43.194638 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:43.194844 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:43.194850 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:43.194890 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 20:43:43.194940 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 20:43:43.195024 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260926 20:43:43.195194 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.188464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54166 (local address 127.0.98.126:39137)
0926 20:43:43.188510 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:43.188513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:43.188648 (+ 135us) server_negotiation.cc:407] Connection header received
0926 20:43:43.188730 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:43.188731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:43.188748 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:43.188810 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:43.189276 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.189803 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.190514 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.190698 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.190784 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.192885 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:43.192887 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:43.192887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:43.192893 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:43.194119 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:43.194448 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:43.194449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:43.194449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:43.194506 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:43.194690 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:43.194691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:43.194691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:43.194784 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:43.194841 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:43.194970 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 20:43:43.195071 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":22}
I20260926 20:43:43.196954 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:54166:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:43:43.197172 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:43:43.198542 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 627d4dc5ae8e4731ba5fc00000e68805 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7043e5940e534f6d84f69beed28ac80d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:43:43.198598 1782 tablet_service.cc:1467] Processing CreateTablet for tablet 627d4dc5ae8e4731ba5fc00000e68805 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7043e5940e534f6d84f69beed28ac80d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:43:43.198654 1922 tablet_service.cc:1467] Processing CreateTablet for tablet 627d4dc5ae8e4731ba5fc00000e68805 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7043e5940e534f6d84f69beed28ac80d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:43:43.198714 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 627d4dc5ae8e4731ba5fc00000e68805. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:43.198768 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 627d4dc5ae8e4731ba5fc00000e68805. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:43.198779 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 627d4dc5ae8e4731ba5fc00000e68805. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:43.199862 2012 tablet_bootstrap.cc:492] T 627d4dc5ae8e4731ba5fc00000e68805 P 8b12c302b49841309fe0f11952573243: Bootstrap starting.
I20260926 20:43:43.200193 2011 tablet_bootstrap.cc:492] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458: Bootstrap starting.
I20260926 20:43:43.200258 2012 tablet_bootstrap.cc:654] T 627d4dc5ae8e4731ba5fc00000e68805 P 8b12c302b49841309fe0f11952573243: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:43.200248 2013 tablet_bootstrap.cc:492] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13: Bootstrap starting.
I20260926 20:43:43.200512 2013 tablet_bootstrap.cc:654] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:43.200654 2011 tablet_bootstrap.cc:654] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:43.200963 2012 tablet_bootstrap.cc:492] T 627d4dc5ae8e4731ba5fc00000e68805 P 8b12c302b49841309fe0f11952573243: No bootstrap required, opened a new log
I20260926 20:43:43.201016 2012 ts_tablet_manager.cc:1397] T 627d4dc5ae8e4731ba5fc00000e68805 P 8b12c302b49841309fe0f11952573243: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:43:43.201185 2012 raft_consensus.cc:348] T 627d4dc5ae8e4731ba5fc00000e68805 P 8b12c302b49841309fe0f11952573243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } }
I20260926 20:43:43.201275 2012 raft_consensus.cc:374] T 627d4dc5ae8e4731ba5fc00000e68805 P 8b12c302b49841309fe0f11952573243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:43.201298 2012 raft_consensus.cc:729] T 627d4dc5ae8e4731ba5fc00000e68805 P 8b12c302b49841309fe0f11952573243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b12c302b49841309fe0f11952573243, State: Initialized, Role: FOLLOWER
I20260926 20:43:43.201308 2011 tablet_bootstrap.cc:492] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458: No bootstrap required, opened a new log
I20260926 20:43:43.201359 2011 ts_tablet_manager.cc:1397] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:43:43.201341 2012 consensus_queue.cc:260] T 627d4dc5ae8e4731ba5fc00000e68805 P 8b12c302b49841309fe0f11952573243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } }
I20260926 20:43:43.201444 2012 ts_tablet_manager.cc:1428] T 627d4dc5ae8e4731ba5fc00000e68805 P 8b12c302b49841309fe0f11952573243: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:43.201502 2011 raft_consensus.cc:348] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } }
I20260926 20:43:43.201495 2018 tablet_replica.cc:439] T 627d4dc5ae8e4731ba5fc00000e68805 P 8b12c302b49841309fe0f11952573243: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } } }
I20260926 20:43:43.201581 2011 raft_consensus.cc:374] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:43.201606 2011 raft_consensus.cc:729] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5c007236ad94556b2bb497f0735b458, State: Initialized, Role: FOLLOWER
I20260926 20:43:43.201592 2018 tablet_replica.cc:442] T 627d4dc5ae8e4731ba5fc00000e68805 P 8b12c302b49841309fe0f11952573243: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:43:43.201649 2011 consensus_queue.cc:260] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } }
I20260926 20:43:43.201750 2011 ts_tablet_manager.cc:1428] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:43.201920 2013 tablet_bootstrap.cc:492] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13: No bootstrap required, opened a new log
I20260926 20:43:43.201963 2013 ts_tablet_manager.cc:1397] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:43:43.201979 2017 tablet_replica.cc:439] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } } }
I20260926 20:43:43.202055 2013 raft_consensus.cc:348] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } }
I20260926 20:43:43.202098 2017 tablet_replica.cc:442] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:43:43.202119 2013 raft_consensus.cc:374] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:43.202147 2013 raft_consensus.cc:729] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771f4702d1024b5ab25fd80b7c7e2d13, State: Initialized, Role: FOLLOWER
I20260926 20:43:43.202193 2013 consensus_queue.cc:260] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } }
I20260926 20:43:43.202306 2013 ts_tablet_manager.cc:1428] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:43.202312 2029 tablet_replica.cc:439] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } } }
I20260926 20:43:43.202440 2029 tablet_replica.cc:442] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:43:43.289400 2017 raft_consensus.cc:482] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:43:43.289500 2017 raft_consensus.cc:504] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } }
I20260926 20:43:43.289969 2017 leader_election.cc:288] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 771f4702d1024b5ab25fd80b7c7e2d13 (127.0.98.65:45995), 8b12c302b49841309fe0f11952573243 (127.0.98.66:41593)
I20260926 20:43:43.293880 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.290296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:41949 (local address 127.0.98.66:41593)
0926 20:43:43.290341 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:43.290343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:43.290502 (+ 159us) server_negotiation.cc:407] Connection header received
0926 20:43:43.290542 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:43.290543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:43.290553 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:43.290611 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:43.291231 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.291852 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.292966 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.293511 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.293603 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.293616 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:43.293759 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 20:43:43.293788 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:43:43.293893 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.290215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:41593 (local address 127.0.98.67:41949)
0926 20:43:43.290396 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:43.290405 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:43.290460 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:43.290632 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:43.290633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:43.290635 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:43.291136 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.291138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.291930 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.291930 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.292737 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.292739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.293605 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.293606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.293684 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.293689 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:43:43.293750 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 20:43:43.293788 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":109,"threads_started":1}
I20260926 20:43:43.294133 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.290146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:45995 (local address 127.0.98.67:52811)
0926 20:43:43.290213 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:43.290226 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:43.290281 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:43.290472 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:43.290474 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:43.290476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:43.290551 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.290553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.291672 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.291674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.293085 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.293088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.293870 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.293871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.293948 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.293951 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:43:43.294012 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 20:43:43.294065 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 20:43:43.294166 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "627d4dc5ae8e4731ba5fc00000e68805" candidate_uuid: "c5c007236ad94556b2bb497f0735b458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b12c302b49841309fe0f11952573243" is_pre_election: true
I20260926 20:43:43.294202 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.290274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:52811 (local address 127.0.98.65:45995)
0926 20:43:43.290322 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:43.290324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:43.290332 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:43:43.290361 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:43.290362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:43.290375 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:43.290441 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:43.290633 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.291580 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.293172 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.293795 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.293863 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.293871 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:43.294091 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 20:43:43.294126 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 20:43:43.294252 1802 raft_consensus.cc:2454] T 627d4dc5ae8e4731ba5fc00000e68805 P 8b12c302b49841309fe0f11952573243 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5c007236ad94556b2bb497f0735b458 in term 0.
I20260926 20:43:43.294409 1876 leader_election.cc:302] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b12c302b49841309fe0f11952573243, c5c007236ad94556b2bb497f0735b458; no voters:
I20260926 20:43:43.294456 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "627d4dc5ae8e4731ba5fc00000e68805" candidate_uuid: "c5c007236ad94556b2bb497f0735b458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" is_pre_election: true
I20260926 20:43:43.294538 2017 raft_consensus.cc:2781] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:43:43.294536 1662 raft_consensus.cc:2454] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5c007236ad94556b2bb497f0735b458 in term 0.
I20260926 20:43:43.294580 2017 raft_consensus.cc:482] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:43:43.294603 2017 raft_consensus.cc:3037] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:43.311291 2017 raft_consensus.cc:504] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } }
I20260926 20:43:43.311509 2017 leader_election.cc:288] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 [CANDIDATE]: Term 1 election: Requested vote from peers 771f4702d1024b5ab25fd80b7c7e2d13 (127.0.98.65:45995), 8b12c302b49841309fe0f11952573243 (127.0.98.66:41593)
I20260926 20:43:43.311782 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "627d4dc5ae8e4731ba5fc00000e68805" candidate_uuid: "c5c007236ad94556b2bb497f0735b458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b12c302b49841309fe0f11952573243"
I20260926 20:43:43.311862 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "627d4dc5ae8e4731ba5fc00000e68805" candidate_uuid: "c5c007236ad94556b2bb497f0735b458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771f4702d1024b5ab25fd80b7c7e2d13"
I20260926 20:43:43.311921 1802 raft_consensus.cc:3037] T 627d4dc5ae8e4731ba5fc00000e68805 P 8b12c302b49841309fe0f11952573243 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:43.311935 1662 raft_consensus.cc:3037] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:43.312711 1802 raft_consensus.cc:2454] T 627d4dc5ae8e4731ba5fc00000e68805 P 8b12c302b49841309fe0f11952573243 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5c007236ad94556b2bb497f0735b458 in term 1.
I20260926 20:43:43.312711 1662 raft_consensus.cc:2454] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5c007236ad94556b2bb497f0735b458 in term 1.
I20260926 20:43:43.312917 1873 leader_election.cc:302] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 771f4702d1024b5ab25fd80b7c7e2d13, c5c007236ad94556b2bb497f0735b458; no voters:
I20260926 20:43:43.313032 2017 raft_consensus.cc:2781] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:43.313215 2017 raft_consensus.cc:686] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 [term 1 LEADER]: Becoming Leader. State: Replica: c5c007236ad94556b2bb497f0735b458, State: Running, Role: LEADER
I20260926 20:43:43.313306 2017 consensus_queue.cc:237] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } }
I20260926 20:43:43.313750 2053 tablet_replica.cc:439] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458: TxnStatusTablet state changed. Reason: New leader c5c007236ad94556b2bb497f0735b458. Latest consensus state: current_term: 1 leader_uuid: "c5c007236ad94556b2bb497f0735b458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } } }
I20260926 20:43:43.313916 2053 tablet_replica.cc:442] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:43:43.314113 2055 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:43:43.314117 1504 catalog_manager.cc:5495] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5c007236ad94556b2bb497f0735b458 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "c5c007236ad94556b2bb497f0735b458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } health_report { overall_health: HEALTHY } } }
I20260926 20:43:43.314239 2055 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260926 20:43:43.320941 1709 tablet.cc:2369] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:43:43.321986 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.319201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46814 (local address 127.0.98.67:34343)
0926 20:43:43.319246 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:43.319248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:43.319271 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:43:43.319395 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:43.319396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:43.319429 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:43.319521 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:43:43.319649 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.320247 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.321091 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.321379 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.321455 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.321649 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:43:43.321754 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:43:43.321829 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:43.321895 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 20:43:43.321913 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 20:43:43.322000 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.319111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:34343 (local address 127.0.0.1:46814)
0926 20:43:43.319214 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:43.319229 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:43.319294 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:43.319520 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:43.319521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:43.319522 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:43:43.319578 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.319581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.320329 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.320330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.320963 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.320965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.321454 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.321455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.321522 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.321546 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:43:43.321829 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:43:43.321839 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 20:43:43.321880 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 20:43:43.321932 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 20:43:43.323237 1802 raft_consensus.cc:1261] T 627d4dc5ae8e4731ba5fc00000e68805 P 8b12c302b49841309fe0f11952573243 [term 1 FOLLOWER]: Refusing update from remote peer c5c007236ad94556b2bb497f0735b458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:43.323395 1662 raft_consensus.cc:1261] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13 [term 1 FOLLOWER]: Refusing update from remote peer c5c007236ad94556b2bb497f0735b458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:43.323506 2017 consensus_queue.cc:1035] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:43:43.323653 2053 consensus_queue.cc:1035] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:43:43.324146 2018 tablet_replica.cc:439] T 627d4dc5ae8e4731ba5fc00000e68805 P 8b12c302b49841309fe0f11952573243: TxnStatusTablet state changed. Reason: New leader c5c007236ad94556b2bb497f0735b458. Latest consensus state: current_term: 1 leader_uuid: "c5c007236ad94556b2bb497f0735b458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } } }
I20260926 20:43:43.324241 2018 tablet_replica.cc:442] T 627d4dc5ae8e4731ba5fc00000e68805 P 8b12c302b49841309fe0f11952573243: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:43:43.324198 2029 tablet_replica.cc:439] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13: TxnStatusTablet state changed. Reason: New leader c5c007236ad94556b2bb497f0735b458. Latest consensus state: current_term: 1 leader_uuid: "c5c007236ad94556b2bb497f0735b458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } } }
I20260926 20:43:43.324283 2029 tablet_replica.cc:442] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:43:43.325124 2017 tablet_replica.cc:439] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c5c007236ad94556b2bb497f0735b458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } } }
I20260926 20:43:43.325209 2017 tablet_replica.cc:442] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:43:43.325443 2029 tablet_replica.cc:439] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c5c007236ad94556b2bb497f0735b458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } } }
I20260926 20:43:43.325528 2029 tablet_replica.cc:442] T 627d4dc5ae8e4731ba5fc00000e68805 P 771f4702d1024b5ab25fd80b7c7e2d13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:43:43.326623 2017 tablet_replica.cc:439] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c5c007236ad94556b2bb497f0735b458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } } }
I20260926 20:43:43.326712 2017 tablet_replica.cc:442] T 627d4dc5ae8e4731ba5fc00000e68805 P c5c007236ad94556b2bb497f0735b458: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:43:43.327271 2018 tablet_replica.cc:439] T 627d4dc5ae8e4731ba5fc00000e68805 P 8b12c302b49841309fe0f11952573243: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c5c007236ad94556b2bb497f0735b458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 } } peers { permanent_uuid: "8b12c302b49841309fe0f11952573243" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41593 } } peers { permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 } } }
I20260926 20:43:43.327366 2018 tablet_replica.cc:442] T 627d4dc5ae8e4731ba5fc00000e68805 P 8b12c302b49841309fe0f11952573243: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:43:43.329937 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.326945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.0.98.66:41593)
0926 20:43:43.327009 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:43.327011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:43.327022 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:43:43.327057 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:43.327058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:43.327085 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:43.327163 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:43:43.327376 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.328214 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.329115 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.329415 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.329488 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.329606 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:43:43.329729 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:43:43.329799 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:43.329834 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 20:43:43.329852 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:43:43.335319 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.332522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:49779 (local address 127.0.98.67:34343)
0926 20:43:43.332597 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:43.332599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:43.332798 (+ 199us) server_negotiation.cc:407] Connection header received
0926 20:43:43.332864 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:43.332865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:43.332873 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:43.332921 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:43:43.333035 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.333536 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.334659 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.334863 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.334925 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.335108 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:43:43.335133 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:43:43.335173 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:43.335250 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 20:43:43.335268 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:43:43.335383 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.332465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:34343 (local address 127.0.98.66:49779)
0926 20:43:43.332704 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:43.332753 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:43.332811 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:43.332932 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:43.332932 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:43.332934 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:43:43.332982 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.332983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.333612 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.333613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.334553 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.334555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.334942 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.334943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.335010 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.335025 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:43:43.335191 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:43:43.335193 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:43:43.335249 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 20:43:43.335290 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":150,"threads_started":1}
I20260926 20:43:43.337735 1937 raft_consensus.cc:1261] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458 [term 1 FOLLOWER]: Refusing update from remote peer 8b12c302b49841309fe0f11952573243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:43.337877 1662 raft_consensus.cc:1261] T c5ac546d4a564afb9c541a113afc177f P 771f4702d1024b5ab25fd80b7c7e2d13 [term 1 FOLLOWER]: Refusing update from remote peer 8b12c302b49841309fe0f11952573243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:43.337987 2018 consensus_queue.cc:1035] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c007236ad94556b2bb497f0735b458" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:43:43.338356 2041 consensus_queue.cc:1035] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "771f4702d1024b5ab25fd80b7c7e2d13" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 45995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:43:43.347419 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.344334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:41593 (local address 127.0.98.67:52955)
0926 20:43:43.344683 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:43.344696 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:43.344753 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:43.344890 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:43.344891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:43.344892 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:43:43.344953 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.344954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.345570 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.345571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.346567 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:43.346570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.347037 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.347038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:43.347097 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.347111 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:43:43.347290 (+ 179us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:43:43.347292 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:43:43.347326 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 20:43:43.347362 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302}
I20260926 20:43:43.347429 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.344378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:52955 (local address 127.0.98.66:41593)
0926 20:43:43.344419 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:43.344421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:43.344744 (+ 323us) server_negotiation.cc:407] Connection header received
0926 20:43:43.344830 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:43.344832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:43.344840 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:43.344884 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:43:43.345021 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.345484 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.346665 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.346962 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.347029 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.347196 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:43:43.347231 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:43:43.347283 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:43.347329 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 20:43:43.347350 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:43:43.357456 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:43:43.357573 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:43:43.358273 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a0a59e088e34c5c8aceaf186c04c2d9: Sending DeleteTablet for 3 replicas of tablet c5ac546d4a564afb9c541a113afc177f
I20260926 20:43:43.358531 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet c5ac546d4a564afb9c541a113afc177f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:43 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:52570
I20260926 20:43:43.358693 2034 tablet_replica.cc:330] T c5ac546d4a564afb9c541a113afc177f P 771f4702d1024b5ab25fd80b7c7e2d13: stopping tablet replica
I20260926 20:43:43.358747 2034 raft_consensus.cc:2229] T c5ac546d4a564afb9c541a113afc177f P 771f4702d1024b5ab25fd80b7c7e2d13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:43:43.358788 2034 raft_consensus.cc:2258] T c5ac546d4a564afb9c541a113afc177f P 771f4702d1024b5ab25fd80b7c7e2d13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:43.358860 1782 tablet_service.cc:1514] Processing DeleteTablet for tablet c5ac546d4a564afb9c541a113afc177f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:43 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:35274
I20260926 20:43:43.358942 2035 tablet_replica.cc:330] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243: stopping tablet replica
I20260926 20:43:43.358984 2035 raft_consensus.cc:2229] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:43:43.359083 2035 raft_consensus.cc:2258] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:43.359131 2034 ts_tablet_manager.cc:1905] T c5ac546d4a564afb9c541a113afc177f P 771f4702d1024b5ab25fd80b7c7e2d13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:43:43.359268 1922 tablet_service.cc:1514] Processing DeleteTablet for tablet c5ac546d4a564afb9c541a113afc177f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:43 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:46804
I20260926 20:43:43.359411 2035 ts_tablet_manager.cc:1905] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:43:43.360853 2036 tablet_replica.cc:330] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458: stopping tablet replica
I20260926 20:43:43.360944 2036 raft_consensus.cc:2229] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:43:43.360989 2036 raft_consensus.cc:2258] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:43.363006 2036 ts_tablet_manager.cc:1905] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 20:43:43 dist-test-slave-4hd3 krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455422, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260926 20:43:43.372119 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.361612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.0.98.65:45995)
0926 20:43:43.361677 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:43.361679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:43.361916 (+ 237us) server_negotiation.cc:407] Connection header received
0926 20:43:43.362046 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:43.362047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:43.362067 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:43.362099 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:43.362836 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.363556 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.364714 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.365005 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.365047 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.368657 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:43.368669 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:43.368669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:43.368678 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:43.370675 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:43.371090 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:43.371094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:43.371094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:43.371155 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:43.371390 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:43.371390 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:43.371391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:43.371556 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:43.371667 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:43.371926 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 20:43:43.372009 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 20:43:43.372756 1702 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52602
I20260926 20:43:43.381376 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.373774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54172 (local address 127.0.98.126:39137)
0926 20:43:43.373823 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:43.373825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:43.373997 (+ 172us) server_negotiation.cc:407] Connection header received
0926 20:43:43.374069 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:43.374070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:43.374081 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:43.374144 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:43.374795 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.375608 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.376664 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:43.376970 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:43.377045 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:43.378479 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:43.378481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:43.378481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:43.378490 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:43.380187 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:43.380566 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:43.380567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:43.380567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:43.380628 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:43.380847 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:43.380847 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:43.380848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:43.380980 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:43.381091 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:43.381201 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 20:43:43.381276 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 20:43:43.381690 1503 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54172
I20260926 20:43:43.382375 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 1577
I20260926 20:43:43.416962 2035 ts_tablet_manager.cc:1918] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:43:43.417094 2035 log.cc:1196] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/c5ac546d4a564afb9c541a113afc177f
I20260926 20:43:43.417351 2035 ts_tablet_manager.cc:1939] T c5ac546d4a564afb9c541a113afc177f P 8b12c302b49841309fe0f11952573243: Deleting consensus metadata
W20260926 20:43:43.417994 1489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:45995 (error 108)
I20260926 20:43:43.418277 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 1717
I20260926 20:43:43.418736 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.418602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:45995 (local address 127.0.0.1:50634)
0926 20:43:43.418652 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:43.418693 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:45995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 20:43:43.418848 1489 catalog_manager.cc:4584] TS 771f4702d1024b5ab25fd80b7c7e2d13 (127.0.98.65:45995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c5ac546d4a564afb9c541a113afc177f: Network error: Client connection negotiation failed: client connection to 127.0.98.65:45995: connect: Connection refused (error 111)
I20260926 20:43:43.432816 2036 ts_tablet_manager.cc:1918] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:43:43.432907 2036 log.cc:1196] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/c5ac546d4a564afb9c541a113afc177f
I20260926 20:43:43.433125 2036 ts_tablet_manager.cc:1939] T c5ac546d4a564afb9c541a113afc177f P c5c007236ad94556b2bb497f0735b458: Deleting consensus metadata
I20260926 20:43:43.434029 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 1857
I20260926 20:43:43.434052 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.433899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:41593 (local address 127.0.0.1:47976)
0926 20:43:43.433975 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:43.434012 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.66:41593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 20:43:43.454046 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 1473
I20260926 20:43:43.454270 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:43.454091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:34343 (local address 127.0.0.1:51584)
0926 20:43:43.454176 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:43.454221 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.67:34343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
2026-09-26T20:43:43Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2370 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:43:43 dist-test-slave-4hd3 krb5kdc[2077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:43:43 dist-test-slave-4hd3 krb5kdc[2077](info): set up 2 sockets
Sep 26 20:43:43 dist-test-slave-4hd3 krb5kdc[2077](info): commencing operation
krb5kdc: starting...
W20260926 20:43:44.835448 393 mini_kdc.cc:121] Time spent starting KDC: real 1.359s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:43:44 dist-test-slave-4hd3 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:43:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:43:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:43:44.955232 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38439
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:38439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:45.034338 2090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:45.034591 2090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:45.034626 2090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:45.035809 2090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:43:45.035881 2090 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:43:45.035903 2090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:45.035925 2090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:45.035954 2090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:43:45.035979 2090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:43:45.037439 2090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:38439
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:45.037719 2090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:45.037946 2090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:45.042601 2098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:45.042615 2095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:45.042601 2096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:45.042985 2090 server_base.cc:1034] running on GCE node
I20260926 20:43:45.043243 2090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:45.043532 2090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:45.044667 2090 hybrid_clock.cc:648] HybridClock initialized: now 1790455425044645 us; error 46 us; skew 500 ppm
Sep 26 20:43:45 dist-test-slave-4hd3 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:45.047106 2090 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:43:45.047595 2090 webserver.cc:460] Webserver started at http://127.0.98.126:43929/ using document root <none> and password file <none>
I20260926 20:43:45.047854 2090 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:45.047904 2090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:45.048023 2090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:45.049060 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "adf28b02cb3348d1b8646f17c2bb025a"
format_stamp: "Formatted at 2026-09-26 20:43:45 on dist-test-slave-4hd3"
server_key: "893f80b18eb19831ea9262a2ec6c7cea"
server_key_iv: "ec579afc72fd76100053d9be4b996942"
server_key_version: "encryptionkey@0"
I20260926 20:43:45.049420 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "adf28b02cb3348d1b8646f17c2bb025a"
format_stamp: "Formatted at 2026-09-26 20:43:45 on dist-test-slave-4hd3"
server_key: "893f80b18eb19831ea9262a2ec6c7cea"
server_key_iv: "ec579afc72fd76100053d9be4b996942"
server_key_version: "encryptionkey@0"
I20260926 20:43:45.050789 2090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:43:45.051605 2105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:45.051834 2090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:43:45.051908 2090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "adf28b02cb3348d1b8646f17c2bb025a"
format_stamp: "Formatted at 2026-09-26 20:43:45 on dist-test-slave-4hd3"
server_key: "893f80b18eb19831ea9262a2ec6c7cea"
server_key_iv: "ec579afc72fd76100053d9be4b996942"
server_key_version: "encryptionkey@0"
I20260926 20:43:45.052001 2090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:45.067888 2090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:45.070063 2090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:45.070235 2090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:45.075050 2090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:38439
I20260926 20:43:45.075090 2167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:38439 every 8 connection(s)
I20260926 20:43:45.075464 2090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:43:45.076192 2168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:45.078588 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a: Bootstrap starting.
I20260926 20:43:45.079131 2168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:45.079363 2168 log.cc:824] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:45.079903 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 2090
I20260926 20:43:45.080005 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:43:45.080163 393 external_mini_cluster.cc:1428] Setting key a315aa9ba49bb21bc0b84888c64656c0
I20260926 20:43:45.080152 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a: No bootstrap required, opened a new log
I20260926 20:43:45.083148 2168 raft_consensus.cc:348] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf28b02cb3348d1b8646f17c2bb025a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38439 } }
I20260926 20:43:45.083324 2168 raft_consensus.cc:374] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:45.083356 2168 raft_consensus.cc:729] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf28b02cb3348d1b8646f17c2bb025a, State: Initialized, Role: FOLLOWER
I20260926 20:43:45.083472 2168 consensus_queue.cc:260] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf28b02cb3348d1b8646f17c2bb025a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38439 } }
I20260926 20:43:45.083540 2168 raft_consensus.cc:388] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:43:45.083578 2168 raft_consensus.cc:482] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:43:45.083621 2168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:45.084290 2168 raft_consensus.cc:504] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf28b02cb3348d1b8646f17c2bb025a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38439 } }
I20260926 20:43:45.084380 2168 leader_election.cc:302] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adf28b02cb3348d1b8646f17c2bb025a; no voters:
I20260926 20:43:45.084573 2168 leader_election.cc:288] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:43:45.084625 2173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:45.084826 2173 raft_consensus.cc:686] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a [term 1 LEADER]: Becoming Leader. State: Replica: adf28b02cb3348d1b8646f17c2bb025a, State: Running, Role: LEADER
I20260926 20:43:45.084846 2168 sys_catalog.cc:564] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:43:45.084918 2173 consensus_queue.cc:237] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf28b02cb3348d1b8646f17c2bb025a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38439 } }
I20260926 20:43:45.085306 2173 sys_catalog.cc:455] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a [sys.catalog]: SysCatalogTable state changed. Reason: New leader adf28b02cb3348d1b8646f17c2bb025a. Latest consensus state: current_term: 1 leader_uuid: "adf28b02cb3348d1b8646f17c2bb025a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf28b02cb3348d1b8646f17c2bb025a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38439 } } }
I20260926 20:43:45.085336 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adf28b02cb3348d1b8646f17c2bb025a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf28b02cb3348d1b8646f17c2bb025a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38439 } } }
I20260926 20:43:45.085462 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:45.085665 2173 sys_catalog.cc:458] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:45.086069 2182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:43:45.086670 2182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 20:43:45 dist-test-slave-4hd3 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:45.088419 2182 catalog_manager.cc:1347] Generated new cluster ID: d0bb1d8fc98a441596905d9c1887469e
I20260926 20:43:45.088466 2182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:43:45.092278 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.080699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39802 (local address 127.0.98.126:38439)
0926 20:43:45.080901 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:45.080911 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:45.080948 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:43:45.081663 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:45.081673 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:45.081739 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:45.081895 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:45.082519 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.083902 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.084833 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.085432 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.085528 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.088506 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:45.088510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:45.088510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:45.088522 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:45.090543 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:45.090945 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:45.090949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:45.090949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:45.091059 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:45.091271 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:45.091272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:45.091272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:45.091593 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:45.091780 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:45.091860 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 20:43:45.092010 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260926 20:43:45.107210 2182 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:43:45.107983 2182 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:43:45.111616 2182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a: Generated new TSK 0
I20260926 20:43:45.111837 2182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260926 20:43:45.138983 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38439
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:45.220952 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:45.221176 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:45.221215 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:45.222348 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:45.222416 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:45.222453 2194 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:43:45.222500 2194 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:43:45.222539 2194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:43:45.224040 2194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:38439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:45.224291 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:45.224514 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:43:45.229562 2194 server_base.cc:1034] running on GCE node
W20260926 20:43:45.229522 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:45.229498 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:45.229498 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:45.230019 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:45.230270 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:45.231436 2194 hybrid_clock.cc:648] HybridClock initialized: now 1790455425231417 us; error 38 us; skew 500 ppm
Sep 26 20:43:45 dist-test-slave-4hd3 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:45.234025 2194 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:43:45.234476 2194 webserver.cc:460] Webserver started at http://127.0.98.65:35987/ using document root <none> and password file <none>
I20260926 20:43:45.234740 2194 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:45.234814 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:45.234968 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:45.236104 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8"
format_stamp: "Formatted at 2026-09-26 20:43:45 on dist-test-slave-4hd3"
server_key: "816fb90b642e2a52f97784df03899222"
server_key_iv: "9b4deba0a4d4fdeef24018e13a5e204c"
server_key_version: "encryptionkey@0"
I20260926 20:43:45.236466 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8"
format_stamp: "Formatted at 2026-09-26 20:43:45 on dist-test-slave-4hd3"
server_key: "816fb90b642e2a52f97784df03899222"
server_key_iv: "9b4deba0a4d4fdeef24018e13a5e204c"
server_key_version: "encryptionkey@0"
I20260926 20:43:45.237838 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:43:45.238648 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:45.238858 2194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:43:45.238981 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8"
format_stamp: "Formatted at 2026-09-26 20:43:45 on dist-test-slave-4hd3"
server_key: "816fb90b642e2a52f97784df03899222"
server_key_iv: "9b4deba0a4d4fdeef24018e13a5e204c"
server_key_version: "encryptionkey@0"
I20260926 20:43:45.239056 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:45.262642 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:45.265094 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:45.265270 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:45.265892 2194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:45.265946 2194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:45.265982 2194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:45.266006 2194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:43:45 dist-test-slave-4hd3 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:45.273455 2194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:43077
I20260926 20:43:45.273545 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:43077 every 8 connection(s)
I20260926 20:43:45.273881 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:43:45.274524 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 2194
I20260926 20:43:45.274626 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:43:45.274821 393 external_mini_cluster.cc:1428] Setting key ab4593214e040078d35daef529a3b808
I20260926 20:43:45.276683 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.266370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:46233 (local address 127.0.98.126:38439)
0926 20:43:45.266443 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:45.266446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:45.266688 (+ 242us) server_negotiation.cc:407] Connection header received
0926 20:43:45.267177 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:45.267179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:45.267199 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:45.267283 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:45.268100 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.268972 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.270124 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.270498 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.270571 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.273247 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:45.273249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:45.273249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:45.273258 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:45.275051 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:45.275507 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:45.275508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:45.275509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:45.275588 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:45.275926 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:45.275927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:45.275928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:45.276087 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:45.276165 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:45.276431 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 20:43:45.276530 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:43:45.276801 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.266343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38439 (local address 127.0.98.65:46233)
0926 20:43:45.266515 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:45.266600 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:45.267084 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:45.267431 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:45.267435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:45.267796 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:45.268010 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.268015 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.269116 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.269117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.270001 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.270003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.270609 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.270610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.270746 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.271314 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:45.271323 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:45.273138 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:45.275164 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:45.275171 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:45.275173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:45.275414 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:45.275671 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:45.275671 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:45.275672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:45.275856 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:45.276157 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:45.276161 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:45.276293 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 20:43:45.276373 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 20:43:45.276507 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260926 20:43:45.278276 2325 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38439
I20260926 20:43:45.278362 2325 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:45.278550 2325 heartbeater.cc:511] Master 127.0.98.126:38439 requested a full tablet report, sending...
I20260926 20:43:45.279112 2121 ts_manager.cc:194] Registered new tserver with Master: 9d3b545cc7fe4139ba6b5f9f8938a0e8 (127.0.98.65:43077)
I20260926 20:43:45.280187 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:46233
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260926 20:43:45.284837 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.278424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:47217 (local address 127.0.98.126:38439)
0926 20:43:45.278518 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:45.278520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:45.278841 (+ 321us) server_negotiation.cc:407] Connection header received
0926 20:43:45.278903 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:45.278904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:45.278914 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:45.278968 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:45.279397 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.280002 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.280959 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.281278 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.281360 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.282311 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:45.282312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:45.282312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:45.282319 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:45.283851 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:45.284160 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:45.284161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:45.284162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:45.284246 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:45.284398 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:45.284399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:45.284399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:45.284495 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:45.284543 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:45.284657 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 20:43:45.284714 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":72}
I20260926 20:43:45.284920 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.278407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38439 (local address 127.0.98.65:47217)
0926 20:43:45.278767 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:45.278777 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:45.278852 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:45.278974 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:45.278975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:45.279224 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:45.279304 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.279306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.280094 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.280095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.280866 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.280868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.281364 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.281365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.281417 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.281903 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:45.281909 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:45.282222 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:45.283927 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:45.283928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:45.283929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:45.284090 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:45.284300 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:45.284301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:45.284301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:45.284354 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:45.284546 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:45.284546 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:45.284580 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 20:43:45.284631 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 20:43:45.284739 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260926 20:43:45.303539 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38439
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:45.385762 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:45.385977 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:45.386018 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:45.387174 2334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:45.387243 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:45.387276 2334 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:43:45.387307 2334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:43:45.387339 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:43:45.388794 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:38439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:45.389061 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:45.389302 2334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:45.394209 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:45.394284 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:45.394224 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:45.394424 2334 server_base.cc:1034] running on GCE node
I20260926 20:43:45.394696 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:45.394941 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:45.396135 2334 hybrid_clock.cc:648] HybridClock initialized: now 1790455425396125 us; error 31 us; skew 500 ppm
Sep 26 20:43:45 dist-test-slave-4hd3 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:45.402102 2334 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:43:45.402587 2334 webserver.cc:460] Webserver started at http://127.0.98.66:41651/ using document root <none> and password file <none>
I20260926 20:43:45.402833 2334 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:45.402882 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:45.403014 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:45.404115 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "5cfe9af3f0c3451789f72baa48468cad"
format_stamp: "Formatted at 2026-09-26 20:43:45 on dist-test-slave-4hd3"
server_key: "d92a51e5e726f3b00da32b1889ab50d7"
server_key_iv: "7f56f132216b1790c1e91b887572f30a"
server_key_version: "encryptionkey@0"
I20260926 20:43:45.404474 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "5cfe9af3f0c3451789f72baa48468cad"
format_stamp: "Formatted at 2026-09-26 20:43:45 on dist-test-slave-4hd3"
server_key: "d92a51e5e726f3b00da32b1889ab50d7"
server_key_iv: "7f56f132216b1790c1e91b887572f30a"
server_key_version: "encryptionkey@0"
I20260926 20:43:45.405768 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:43:45.406481 2349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:45.406787 2334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:43:45.406859 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "5cfe9af3f0c3451789f72baa48468cad"
format_stamp: "Formatted at 2026-09-26 20:43:45 on dist-test-slave-4hd3"
server_key: "d92a51e5e726f3b00da32b1889ab50d7"
server_key_iv: "7f56f132216b1790c1e91b887572f30a"
server_key_version: "encryptionkey@0"
I20260926 20:43:45.406949 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:45.419523 2334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:45.421550 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:45.421686 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:45.422226 2334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:45.422266 2334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:45.422286 2334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:45.422299 2334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:43:45 dist-test-slave-4hd3 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:45.429539 2334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:37861
I20260926 20:43:45.429639 2464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:37861 every 8 connection(s)
I20260926 20:43:45.429914 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:43:45.432145 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.422638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:34765 (local address 127.0.98.126:38439)
0926 20:43:45.422712 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:45.422715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:45.422934 (+ 219us) server_negotiation.cc:407] Connection header received
0926 20:43:45.423382 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:45.423384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:45.423398 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:45.423480 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:45.424170 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.424722 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.425702 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.426007 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.426108 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.429179 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:45.429181 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:45.429181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:45.429189 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:45.430753 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:45.431160 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:45.431162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:45.431162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:45.431224 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:45.431488 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:45.431488 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:45.431489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:45.431604 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:45.431680 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:45.431967 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 20:43:45.432045 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:43:45.432317 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.422623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38439 (local address 127.0.98.66:34765)
0926 20:43:45.422792 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:45.422829 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:45.423282 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:45.423588 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:45.423592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:45.423871 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:45.424047 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.424051 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.424817 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.424818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.425610 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.425612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.426116 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.426117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.426259 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.426771 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:45.426781 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:45.429098 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:45.430851 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:45.430857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:45.430859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:45.431082 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:45.431292 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:45.431293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:45.431293 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:45.431412 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:45.431670 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:45.431676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:45.431837 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 20:43:45.431947 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 20:43:45.432101 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":107,"threads_started":1}
I20260926 20:43:45.432822 2465 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38439
I20260926 20:43:45.432907 2465 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:45.433112 2465 heartbeater.cc:511] Master 127.0.98.126:38439 requested a full tablet report, sending...
I20260926 20:43:45.433439 2122 ts_manager.cc:194] Registered new tserver with Master: 5cfe9af3f0c3451789f72baa48468cad (127.0.98.66:37861)
I20260926 20:43:45.433950 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:34765
I20260926 20:43:45.438999 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.433818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:40165 (local address 127.0.98.126:38439)
0926 20:43:45.434086 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:45.434088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:45.434096 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:43:45.434113 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:45.434114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:45.434123 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:45.434172 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:45.434480 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.434935 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.435649 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.435920 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.436001 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.436763 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:45.436764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:45.436764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:45.436771 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:45.437893 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:45.438220 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:45.438221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:45.438222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:45.438273 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:45.438477 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:45.438477 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:45.438478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:45.438617 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:45.438695 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:45.438846 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 20:43:45.438915 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243}
I20260926 20:43:45.439007 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.433732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38439 (local address 127.0.98.66:40165)
0926 20:43:45.433875 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:45.433885 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:45.433935 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:45.434178 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:45.434179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:45.434350 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:45.434396 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.434397 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.435025 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.435026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.435549 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.435551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.436003 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.436004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.436081 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.436424 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:45.436427 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:45.436678 (+ 251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:45.437978 (+ 1300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:45.437979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:45.437979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:45.438147 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:45.438348 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:45.438349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:45.438350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:45.438405 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:45.438687 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:45.438688 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:45.438732 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 20:43:45.438819 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 20:43:45.438914 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260926 20:43:45.439354 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 2334
I20260926 20:43:45.439433 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:43:45.439620 393 external_mini_cluster.cc:1428] Setting key f3007bcfcd0cd99a27890132a3817afd
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260926 20:43:45.464480 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38439
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:45.553949 2474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:45.554278 2474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:45.554320 2474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:45.556402 2474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:45.556469 2474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:45.556504 2474 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:43:45.556530 2474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:43:45.556565 2474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:43:45.558851 2474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:38439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:45.559121 2474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:45.559402 2474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:45.566118 2482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:45.566119 2479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:45.566413 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:45.566612 2474 server_base.cc:1034] running on GCE node
I20260926 20:43:45.566840 2474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:45.567091 2474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:45.568279 2474 hybrid_clock.cc:648] HybridClock initialized: now 1790455425568265 us; error 32 us; skew 500 ppm
Sep 26 20:43:45 dist-test-slave-4hd3 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:45.571453 2474 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:43:45.572084 2474 webserver.cc:460] Webserver started at http://127.0.98.67:42855/ using document root <none> and password file <none>
I20260926 20:43:45.572397 2474 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:45.572474 2474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:45.572626 2474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:45.574155 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "d3bd7e5e06c147d88277f2c9364a0e5e"
format_stamp: "Formatted at 2026-09-26 20:43:45 on dist-test-slave-4hd3"
server_key: "b0ce71af25afb910537eb30de9938a02"
server_key_iv: "2e3133ccf51f1334e28e61d6844f7dae"
server_key_version: "encryptionkey@0"
I20260926 20:43:45.574606 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "d3bd7e5e06c147d88277f2c9364a0e5e"
format_stamp: "Formatted at 2026-09-26 20:43:45 on dist-test-slave-4hd3"
server_key: "b0ce71af25afb910537eb30de9938a02"
server_key_iv: "2e3133ccf51f1334e28e61d6844f7dae"
server_key_version: "encryptionkey@0"
I20260926 20:43:45.576555 2474 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 20:43:45.577550 2489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:45.577740 2474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:43:45.577847 2474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "d3bd7e5e06c147d88277f2c9364a0e5e"
format_stamp: "Formatted at 2026-09-26 20:43:45 on dist-test-slave-4hd3"
server_key: "b0ce71af25afb910537eb30de9938a02"
server_key_iv: "2e3133ccf51f1334e28e61d6844f7dae"
server_key_version: "encryptionkey@0"
I20260926 20:43:45.577929 2474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:45.599104 2474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:45.601146 2474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:45.601312 2474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:45.601820 2474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:45.601886 2474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:45.601933 2474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:45.601956 2474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:45.609100 2474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:43661
I20260926 20:43:45.609197 2604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:43661 every 8 connection(s)
Sep 26 20:43:45 dist-test-slave-4hd3 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:45.609516 2474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:43:45.612804 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.602352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:36217 (local address 127.0.98.126:38439)
0926 20:43:45.602425 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:45.602428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:45.602631 (+ 203us) server_negotiation.cc:407] Connection header received
0926 20:43:45.603113 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:45.603114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:45.603129 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:45.603212 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:45.604095 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.604884 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.606615 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.606943 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.607040 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.610106 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:45.610107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:45.610107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:45.610116 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:45.611329 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:45.611844 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:45.611846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:45.611846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:45.611912 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:45.612213 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:45.612214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:45.612214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:45.612326 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:45.612385 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:45.612662 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 20:43:45.612735 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:43:45.612919 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.602322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38439 (local address 127.0.98.67:36217)
0926 20:43:45.602500 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:45.602541 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:45.603014 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:45.603305 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:45.603310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:45.603683 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:45.604004 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.604012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.605007 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.605009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.606504 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.606507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.607068 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.607069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.607229 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.607985 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:45.607997 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:45.610006 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:45.611401 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:45.611407 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:45.611408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:45.611743 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:45.611986 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:45.611986 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:45.611987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:45.612139 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:45.612388 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:45.612392 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:45.612505 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 20:43:45.612582 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:43:45.612740 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":113,"threads_started":1}
I20260926 20:43:45.613382 2605 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38439
I20260926 20:43:45.613462 2605 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:45.613649 2605 heartbeater.cc:511] Master 127.0.98.126:38439 requested a full tablet report, sending...
I20260926 20:43:45.614009 2121 ts_manager.cc:194] Registered new tserver with Master: d3bd7e5e06c147d88277f2c9364a0e5e (127.0.98.67:43661)
I20260926 20:43:45.614507 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:36217
I20260926 20:43:45.619251 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 2474
I20260926 20:43:45.619364 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:43:45.619576 393 external_mini_cluster.cc:1428] Setting key 9ae45b850f85933a79549927c3b9a028
I20260926 20:43:45.620289 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.614509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:52511 (local address 127.0.98.126:38439)
0926 20:43:45.614551 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:45.614553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:45.614787 (+ 234us) server_negotiation.cc:407] Connection header received
0926 20:43:45.614806 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:45.614806 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:45.614814 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:45.614891 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:45.615206 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.615697 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.616416 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.616625 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.616696 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.617643 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:45.617644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:45.617644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:45.617651 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:45.619082 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:45.619469 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:45.619471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:45.619471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:45.619542 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:45.619840 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:45.619841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:45.619841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:45.619943 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:45.619995 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:45.620155 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 20:43:45.620230 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:43:45.620332 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.614429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38439 (local address 127.0.98.67:52511)
0926 20:43:45.614597 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:45.614608 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:45.614671 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:45.614895 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:45.614896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:45.615074 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:45.615126 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.615128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.615815 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.615815 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.616333 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.616335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.616708 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.616709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.616791 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.617250 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:45.617254 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:45.617529 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:45.619188 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:45.619189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:45.619190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:45.619380 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:45.619651 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:45.619668 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:45.619668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:45.619762 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:45.619990 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:45.619991 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:45.620041 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 20:43:45.620110 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:43:45.620243 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
I20260926 20:43:45.620507 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:43:45 dist-test-slave-4hd3 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:43:45 dist-test-slave-4hd3 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:45.643515 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.636425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39812 (local address 127.0.98.126:38439)
0926 20:43:45.636484 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:45.636490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:45.636501 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:43:45.636527 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:45.636528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:45.636537 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:45.636603 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:45.637186 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.637666 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.638392 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.638610 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.638677 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.640984 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:45.640985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:45.640985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:45.640992 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:45.642484 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:45.642835 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:45.642837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:45.642837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:45.642892 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:45.643073 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:45.643074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:45.643074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:45.643179 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:45.643250 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:45.643349 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 20:43:45.643406 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:43:45.645380 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:43:45.646868 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:43:45.653630 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.649026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43661 (local address 127.0.0.1:56142)
0926 20:43:45.649179 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:45.649224 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:45.649309 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:45.649636 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:45.649640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:45.649642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:45.649709 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.649711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.651068 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.651069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.652622 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.652625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.653309 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.653310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.653399 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.653413 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 20:43:45.653480 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 20:43:45.653534 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":177,"thread_start_us":51,"threads_started":1}
I20260926 20:43:45.653687 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.649050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43077 (local address 127.0.0.1:42946)
0926 20:43:45.649472 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:45.649484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:45.649534 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:45.649934 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:45.649935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:45.649937 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:45.650010 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.650012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.651186 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.651187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.652539 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.652542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.653452 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.653453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.653507 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.653510 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:43:45.653560 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 20:43:45.653590 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":64,"thread_start_us":81,"threads_started":1}
I20260926 20:43:45.653873 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.649516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58912 (local address 127.0.98.66:37861)
0926 20:43:45.649700 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:45.649704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:45.649719 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:43:45.649772 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:45.649774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:45.649790 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:45.649862 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:45.650175 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.651130 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.652704 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.653391 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.653451 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.653505 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:45.653743 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 20:43:45.653787 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":105,"threads_started":1}
I20260926 20:43:45.653911 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.649300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56142 (local address 127.0.98.67:43661)
0926 20:43:45.649450 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:45.649456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:45.649467 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:43:45.649510 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:45.649512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:45.649529 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:45.649609 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:45.649805 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.650971 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.652696 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.653206 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.653298 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.653359 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:45.653753 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 20:43:45.653799 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260926 20:43:45.654022 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.649034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37861 (local address 127.0.0.1:58912)
0926 20:43:45.649416 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:45.649425 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:45.649462 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:45.649930 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:45.649931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:45.649933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:45.650085 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.650086 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.651257 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.651258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.652584 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.652587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.653541 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.653542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.653603 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.653607 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:43:45.653658 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 20:43:45.653701 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260926 20:43:45.654392 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.649308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42946 (local address 127.0.98.65:43077)
0926 20:43:45.649765 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:45.649771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:45.649780 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:43:45.649817 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:45.649819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:45.649834 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:45.649975 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:45.650081 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.651091 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.652643 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.653364 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.653451 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.653504 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:45.653569 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 20:43:45.653608 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":86,"threads_started":1}
I20260926 20:43:45.654706 2539 tablet_service.cc:1467] Processing CreateTablet for tablet e5ca7716c20c4e93bf89cc8c588443cb (DEFAULT_TABLE table=test-table [id=b977dc35076748939dfec4276d4bf747]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:45.655062 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ca7716c20c4e93bf89cc8c588443cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:45.655011 2259 tablet_service.cc:1467] Processing CreateTablet for tablet e5ca7716c20c4e93bf89cc8c588443cb (DEFAULT_TABLE table=test-table [id=b977dc35076748939dfec4276d4bf747]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:45.655325 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ca7716c20c4e93bf89cc8c588443cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:45.656340 2399 tablet_service.cc:1467] Processing CreateTablet for tablet e5ca7716c20c4e93bf89cc8c588443cb (DEFAULT_TABLE table=test-table [id=b977dc35076748939dfec4276d4bf747]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:45.656754 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ca7716c20c4e93bf89cc8c588443cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:45.658607 2628 tablet_bootstrap.cc:492] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e: Bootstrap starting.
I20260926 20:43:45.658771 2629 tablet_bootstrap.cc:492] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Bootstrap starting.
I20260926 20:43:45.659358 2628 tablet_bootstrap.cc:654] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:45.659421 2629 tablet_bootstrap.cc:654] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:45.659588 2628 log.cc:824] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:45.659839 2629 log.cc:824] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:45.660315 2632 tablet_bootstrap.cc:492] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad: Bootstrap starting.
I20260926 20:43:45.660560 2628 tablet_bootstrap.cc:492] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e: No bootstrap required, opened a new log
I20260926 20:43:45.660640 2628 ts_tablet_manager.cc:1397] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:43:45.660880 2629 tablet_bootstrap.cc:492] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8: No bootstrap required, opened a new log
I20260926 20:43:45.660910 2632 tablet_bootstrap.cc:654] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:45.660956 2629 ts_tablet_manager.cc:1397] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:43:45.661227 2632 log.cc:824] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:45.662006 2632 tablet_bootstrap.cc:492] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad: No bootstrap required, opened a new log
I20260926 20:43:45.662073 2632 ts_tablet_manager.cc:1397] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:43:45.662405 2628 raft_consensus.cc:348] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } } peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } }
I20260926 20:43:45.662554 2628 raft_consensus.cc:374] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:45.662597 2628 raft_consensus.cc:729] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3bd7e5e06c147d88277f2c9364a0e5e, State: Initialized, Role: FOLLOWER
I20260926 20:43:45.662703 2628 consensus_queue.cc:260] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } } peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } }
I20260926 20:43:45.662972 2628 ts_tablet_manager.cc:1428] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 20:43:45.663005 2605 heartbeater.cc:503] Master 127.0.98.126:38439 was elected leader, sending a full tablet report...
I20260926 20:43:45.663329 2629 raft_consensus.cc:348] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } } peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } }
I20260926 20:43:45.663494 2629 raft_consensus.cc:374] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:45.663527 2629 raft_consensus.cc:729] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d3b545cc7fe4139ba6b5f9f8938a0e8, State: Initialized, Role: FOLLOWER
I20260926 20:43:45.663637 2629 consensus_queue.cc:260] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } } peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } }
I20260926 20:43:45.664057 2629 ts_tablet_manager.cc:1428] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:43:45.664178 2325 heartbeater.cc:503] Master 127.0.98.126:38439 was elected leader, sending a full tablet report...
I20260926 20:43:45.664077 2632 raft_consensus.cc:348] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } } peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } }
I20260926 20:43:45.664475 2632 raft_consensus.cc:374] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:45.664526 2632 raft_consensus.cc:729] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cfe9af3f0c3451789f72baa48468cad, State: Initialized, Role: FOLLOWER
I20260926 20:43:45.664633 2632 consensus_queue.cc:260] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } } peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } }
I20260926 20:43:45.664901 2632 ts_tablet_manager.cc:1428] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:43:45.664940 2465 heartbeater.cc:503] Master 127.0.98.126:38439 was elected leader, sending a full tablet report...
W20260926 20:43:45.680487 2466 tablet.cc:2369] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:43:45.683997 2636 raft_consensus.cc:482] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:43:45.684124 2636 raft_consensus.cc:504] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } } peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } }
I20260926 20:43:45.684549 2636 leader_election.cc:288] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3bd7e5e06c147d88277f2c9364a0e5e (127.0.98.67:43661), 9d3b545cc7fe4139ba6b5f9f8938a0e8 (127.0.98.65:43077)
I20260926 20:43:45.688568 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.684756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:56979 (local address 127.0.98.65:43077)
0926 20:43:45.684806 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:45.684808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:45.684983 (+ 175us) server_negotiation.cc:407] Connection header received
0926 20:43:45.685042 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:45.685043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:45.685053 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:45.685096 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:45.685234 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.686009 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.687565 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.688150 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.688233 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.688243 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:45.688462 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 20:43:45.688500 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 20:43:45.688573 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.684746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:57347 (local address 127.0.98.67:43661)
0926 20:43:45.684788 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:45.684790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:45.684824 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:43:45.684924 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:45.684927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:45.684941 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:45.685010 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:45.685179 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.686009 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.687565 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.688164 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.688233 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.688243 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:45.688462 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 20:43:45.688500 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:43:45.688593 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.684707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43661 (local address 127.0.98.66:57347)
0926 20:43:45.684770 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:45.684781 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:45.684839 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:45.685026 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:45.685027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:45.685029 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:45.685104 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.685105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.686107 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.686108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.687432 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.687435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.688269 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.688271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.688364 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.688370 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:43:45.688443 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 20:43:45.688500 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":43}
I20260926 20:43:45.688869 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.684707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43077 (local address 127.0.98.66:56979)
0926 20:43:45.684889 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:45.684910 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:45.684970 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:45.685112 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:45.685112 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:45.685113 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:45.685175 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.685176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.686107 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.686108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.687435 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.687436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.688270 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.688271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.688364 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.688370 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:43:45.688443 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 20:43:45.688499 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":133,"thread_start_us":95,"threads_started":1}
I20260926 20:43:45.688977 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ca7716c20c4e93bf89cc8c588443cb" candidate_uuid: "5cfe9af3f0c3451789f72baa48468cad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" is_pre_election: true
I20260926 20:43:45.689172 2559 raft_consensus.cc:2454] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cfe9af3f0c3451789f72baa48468cad in term 0.
I20260926 20:43:45.689248 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ca7716c20c4e93bf89cc8c588443cb" candidate_uuid: "5cfe9af3f0c3451789f72baa48468cad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" is_pre_election: true
I20260926 20:43:45.689388 2279 raft_consensus.cc:2454] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cfe9af3f0c3451789f72baa48468cad in term 0.
I20260926 20:43:45.689394 2353 leader_election.cc:302] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cfe9af3f0c3451789f72baa48468cad, d3bd7e5e06c147d88277f2c9364a0e5e; no voters:
I20260926 20:43:45.689555 2636 raft_consensus.cc:2781] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:43:45.689617 2636 raft_consensus.cc:482] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:43:45.689641 2636 raft_consensus.cc:3037] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:45.690493 2636 raft_consensus.cc:504] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } } peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } }
I20260926 20:43:45.690627 2636 leader_election.cc:288] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [CANDIDATE]: Term 1 election: Requested vote from peers d3bd7e5e06c147d88277f2c9364a0e5e (127.0.98.67:43661), 9d3b545cc7fe4139ba6b5f9f8938a0e8 (127.0.98.65:43077)
I20260926 20:43:45.690766 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ca7716c20c4e93bf89cc8c588443cb" candidate_uuid: "5cfe9af3f0c3451789f72baa48468cad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e"
I20260926 20:43:45.690845 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ca7716c20c4e93bf89cc8c588443cb" candidate_uuid: "5cfe9af3f0c3451789f72baa48468cad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8"
I20260926 20:43:45.690855 2559 raft_consensus.cc:3037] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:45.690907 2279 raft_consensus.cc:3037] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:45.691744 2559 raft_consensus.cc:2454] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cfe9af3f0c3451789f72baa48468cad in term 1.
I20260926 20:43:45.691744 2279 raft_consensus.cc:2454] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cfe9af3f0c3451789f72baa48468cad in term 1.
I20260926 20:43:45.691900 2353 leader_election.cc:302] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cfe9af3f0c3451789f72baa48468cad, d3bd7e5e06c147d88277f2c9364a0e5e; no voters:
I20260926 20:43:45.692032 2636 raft_consensus.cc:2781] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:45.692251 2636 raft_consensus.cc:686] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [term 1 LEADER]: Becoming Leader. State: Replica: 5cfe9af3f0c3451789f72baa48468cad, State: Running, Role: LEADER
I20260926 20:43:45.692353 2636 consensus_queue.cc:237] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } } peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } }
I20260926 20:43:45.693111 2122 catalog_manager.cc:5495] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cfe9af3f0c3451789f72baa48468cad (127.0.98.66). New cstate: current_term: 1 leader_uuid: "5cfe9af3f0c3451789f72baa48468cad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } health_report { overall_health: UNKNOWN } } }
Sep 26 20:43:45 dist-test-slave-4hd3 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:45.715332 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.708389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38439 (local address 127.0.0.1:39820)
0926 20:43:45.708589 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:45.708599 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:45.708662 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:45.708791 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:45.708792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:45.709078 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:45.709134 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.709136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.709857 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.709858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.710381 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:45.710382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.710753 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.710758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:45.710807 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.711273 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:45.711277 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:45.712653 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:45.714462 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:45.714467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:45.714469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:45.714654 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:45.714812 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:45.714812 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:45.714813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:45.714858 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:45.715053 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:45.715058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:45.715105 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 20:43:45.715164 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 20:43:45.715256 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":3,"thread_start_us":77,"threads_started":1}
I20260926 20:43:45.715404 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:45.708486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39820 (local address 127.0.98.126:38439)
0926 20:43:45.708578 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:45.708581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:45.708698 (+ 117us) server_negotiation.cc:407] Connection header received
0926 20:43:45.708720 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:45.708721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:45.708730 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:45.708776 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:45.709236 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.709781 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.710461 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:45.710649 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:45.710729 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:45.712753 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:45.712754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:45.712754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:45.712788 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:45.714397 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:45.714719 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:45.714720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:45.714720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:45.714770 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:45.714905 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:45.714906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:45.714906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:45.715000 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:45.715048 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:45.715182 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 20:43:45.715296 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":59}
I20260926 20:43:45.717525 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:39820:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:43:45.717707 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:43:45.719252 2399 tablet_service.cc:1467] Processing CreateTablet for tablet d9cffb5708e84c0b8bedb11731837cef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=38a416b95ce04f00a42e433c556e62e3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:43:45.719271 2259 tablet_service.cc:1467] Processing CreateTablet for tablet d9cffb5708e84c0b8bedb11731837cef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=38a416b95ce04f00a42e433c556e62e3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:43:45.719362 2539 tablet_service.cc:1467] Processing CreateTablet for tablet d9cffb5708e84c0b8bedb11731837cef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=38a416b95ce04f00a42e433c556e62e3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:43:45.719426 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9cffb5708e84c0b8bedb11731837cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:45.719444 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9cffb5708e84c0b8bedb11731837cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:45.719496 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9cffb5708e84c0b8bedb11731837cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:45.720800 2629 tablet_bootstrap.cc:492] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Bootstrap starting.
I20260926 20:43:45.720920 2628 tablet_bootstrap.cc:492] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e: Bootstrap starting.
I20260926 20:43:45.720920 2632 tablet_bootstrap.cc:492] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad: Bootstrap starting.
I20260926 20:43:45.721369 2632 tablet_bootstrap.cc:654] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:45.721437 2629 tablet_bootstrap.cc:654] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:45.721454 2628 tablet_bootstrap.cc:654] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:45.722074 2628 tablet_bootstrap.cc:492] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e: No bootstrap required, opened a new log
I20260926 20:43:45.722095 2632 tablet_bootstrap.cc:492] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad: No bootstrap required, opened a new log
I20260926 20:43:45.722137 2632 ts_tablet_manager.cc:1397] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:43:45.722158 2628 ts_tablet_manager.cc:1397] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:43:45.722306 2628 raft_consensus.cc:348] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:45.722303 2632 raft_consensus.cc:348] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:45.722381 2632 raft_consensus.cc:374] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:45.722383 2628 raft_consensus.cc:374] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:45.722406 2628 raft_consensus.cc:729] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3bd7e5e06c147d88277f2c9364a0e5e, State: Initialized, Role: FOLLOWER
I20260926 20:43:45.722395 2632 raft_consensus.cc:729] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cfe9af3f0c3451789f72baa48468cad, State: Initialized, Role: FOLLOWER
I20260926 20:43:45.722458 2632 consensus_queue.cc:260] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:45.722460 2628 consensus_queue.cc:260] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:45.722550 2628 ts_tablet_manager.cc:1428] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:45.722568 2632 ts_tablet_manager.cc:1428] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:43:45.722633 2634 tablet_replica.cc:439] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } } }
I20260926 20:43:45.722725 2634 tablet_replica.cc:442] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:43:45.722832 2636 tablet_replica.cc:439] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } } }
I20260926 20:43:45.722911 2636 tablet_replica.cc:442] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:43:45.723114 2629 tablet_bootstrap.cc:492] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8: No bootstrap required, opened a new log
I20260926 20:43:45.723191 2629 ts_tablet_manager.cc:1397] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:43:45.723337 2629 raft_consensus.cc:348] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:45.723418 2629 raft_consensus.cc:374] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:45.723450 2629 raft_consensus.cc:729] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d3b545cc7fe4139ba6b5f9f8938a0e8, State: Initialized, Role: FOLLOWER
I20260926 20:43:45.723505 2629 consensus_queue.cc:260] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:45.723591 2629 ts_tablet_manager.cc:1428] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:45.723664 2635 tablet_replica.cc:439] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } } }
I20260926 20:43:45.723790 2635 tablet_replica.cc:442] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 20:43:45.774682 2326 tablet.cc:2369] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:43:45.860201 2606 tablet.cc:2369] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:43:45.952792 2636 raft_consensus.cc:482] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:43:45.952900 2636 raft_consensus.cc:504] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:45.953085 2636 leader_election.cc:288] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d3b545cc7fe4139ba6b5f9f8938a0e8 (127.0.98.65:43077), d3bd7e5e06c147d88277f2c9364a0e5e (127.0.98.67:43661)
I20260926 20:43:45.953423 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9cffb5708e84c0b8bedb11731837cef" candidate_uuid: "5cfe9af3f0c3451789f72baa48468cad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" is_pre_election: true
I20260926 20:43:45.953437 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9cffb5708e84c0b8bedb11731837cef" candidate_uuid: "5cfe9af3f0c3451789f72baa48468cad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" is_pre_election: true
I20260926 20:43:45.953542 2279 raft_consensus.cc:2454] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cfe9af3f0c3451789f72baa48468cad in term 0.
I20260926 20:43:45.953545 2559 raft_consensus.cc:2454] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cfe9af3f0c3451789f72baa48468cad in term 0.
I20260926 20:43:45.953750 2353 leader_election.cc:302] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cfe9af3f0c3451789f72baa48468cad, d3bd7e5e06c147d88277f2c9364a0e5e; no voters:
I20260926 20:43:45.953879 2636 raft_consensus.cc:2781] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:43:45.953955 2636 raft_consensus.cc:482] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:43:45.953984 2636 raft_consensus.cc:3037] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:45.954764 2636 raft_consensus.cc:504] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:45.954908 2636 leader_election.cc:288] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad [CANDIDATE]: Term 1 election: Requested vote from peers 9d3b545cc7fe4139ba6b5f9f8938a0e8 (127.0.98.65:43077), d3bd7e5e06c147d88277f2c9364a0e5e (127.0.98.67:43661)
I20260926 20:43:45.955090 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9cffb5708e84c0b8bedb11731837cef" candidate_uuid: "5cfe9af3f0c3451789f72baa48468cad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8"
I20260926 20:43:45.955127 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9cffb5708e84c0b8bedb11731837cef" candidate_uuid: "5cfe9af3f0c3451789f72baa48468cad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e"
I20260926 20:43:45.955188 2279 raft_consensus.cc:3037] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:45.955189 2559 raft_consensus.cc:3037] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:45.955955 2279 raft_consensus.cc:2454] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cfe9af3f0c3451789f72baa48468cad in term 1.
I20260926 20:43:45.956063 2559 raft_consensus.cc:2454] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cfe9af3f0c3451789f72baa48468cad in term 1.
I20260926 20:43:45.956190 2352 leader_election.cc:302] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cfe9af3f0c3451789f72baa48468cad, 9d3b545cc7fe4139ba6b5f9f8938a0e8; no voters:
I20260926 20:43:45.956295 2636 raft_consensus.cc:2781] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:45.956349 2636 raft_consensus.cc:686] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad [term 1 LEADER]: Becoming Leader. State: Replica: 5cfe9af3f0c3451789f72baa48468cad, State: Running, Role: LEADER
I20260926 20:43:45.956394 2636 consensus_queue.cc:237] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:45.956671 2638 tablet_replica.cc:439] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad: TxnStatusTablet state changed. Reason: New leader 5cfe9af3f0c3451789f72baa48468cad. Latest consensus state: current_term: 1 leader_uuid: "5cfe9af3f0c3451789f72baa48468cad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } } }
I20260926 20:43:45.956765 2638 tablet_replica.cc:442] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:43:45.956970 2650 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:43:45.957104 2650 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 20:43:45.957048 2122 catalog_manager.cc:5495] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cfe9af3f0c3451789f72baa48468cad (127.0.98.66). New cstate: current_term: 1 leader_uuid: "5cfe9af3f0c3451789f72baa48468cad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } health_report { overall_health: UNKNOWN } } }
I20260926 20:43:46.006335 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:46.003798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37861 (local address 127.0.0.1:58920)
0926 20:43:46.003885 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:46.003896 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:46.003949 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:46.004131 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:46.004133 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:46.004134 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:43:46.004187 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:46.004190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.004814 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.004815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:46.005404 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:46.005405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.005789 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.005791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:46.005859 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:46.005877 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:43:46.006164 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:43:46.006173 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 20:43:46.006215 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 20:43:46.006264 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 20:43:46.006331 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:46.003849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58920 (local address 127.0.98.66:37861)
0926 20:43:46.003902 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:46.003905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:46.003931 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:43:46.004032 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:46.004033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:46.004056 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:46.004118 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:43:46.004255 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.004732 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.005493 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.005704 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.005778 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:46.005972 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:43:46.006080 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:43:46.006177 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:46.006220 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 20:43:46.006241 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 20:43:46.008076 2279 raft_consensus.cc:1261] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 1 FOLLOWER]: Refusing update from remote peer 5cfe9af3f0c3451789f72baa48468cad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:46.008076 2559 raft_consensus.cc:1261] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e [term 1 FOLLOWER]: Refusing update from remote peer 5cfe9af3f0c3451789f72baa48468cad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:46.008424 2636 consensus_queue.cc:1035] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:43:46.008667 2636 consensus_queue.cc:1035] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:43:46.009720 2635 tablet_replica.cc:439] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8: TxnStatusTablet state changed. Reason: New leader 5cfe9af3f0c3451789f72baa48468cad. Latest consensus state: current_term: 1 leader_uuid: "5cfe9af3f0c3451789f72baa48468cad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } } }
I20260926 20:43:46.009856 2635 tablet_replica.cc:442] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:43:46.009910 2634 tablet_replica.cc:439] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e: TxnStatusTablet state changed. Reason: New leader 5cfe9af3f0c3451789f72baa48468cad. Latest consensus state: current_term: 1 leader_uuid: "5cfe9af3f0c3451789f72baa48468cad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } } }
I20260926 20:43:46.010027 2634 tablet_replica.cc:442] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:43:46.010939 2636 tablet_replica.cc:439] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cfe9af3f0c3451789f72baa48468cad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } } }
I20260926 20:43:46.011037 2636 tablet_replica.cc:442] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:43:46.010939 2653 mvcc.cc:205] Tried to move back new op lower bound from 7333705424926265344 to 7333705424717783040. Current Snapshot: MvccSnapshot[applied={T|T < 7333705424926265344}]
I20260926 20:43:46.011138 2654 mvcc.cc:205] Tried to move back new op lower bound from 7333705424926265344 to 7333705424717783040. Current Snapshot: MvccSnapshot[applied={T|T < 7333705424926265344}]
I20260926 20:43:46.011116 2635 tablet_replica.cc:439] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cfe9af3f0c3451789f72baa48468cad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } } }
I20260926 20:43:46.011315 2635 tablet_replica.cc:442] T d9cffb5708e84c0b8bedb11731837cef P 9d3b545cc7fe4139ba6b5f9f8938a0e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:43:46.011535 2638 tablet_replica.cc:439] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cfe9af3f0c3451789f72baa48468cad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } } }
I20260926 20:43:46.011615 2638 tablet_replica.cc:442] T d9cffb5708e84c0b8bedb11731837cef P 5cfe9af3f0c3451789f72baa48468cad: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:43:46.012235 2634 tablet_replica.cc:439] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cfe9af3f0c3451789f72baa48468cad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } } }
I20260926 20:43:46.012324 2634 tablet_replica.cc:442] T d9cffb5708e84c0b8bedb11731837cef P d3bd7e5e06c147d88277f2c9364a0e5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:43:46.016547 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:46.013921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58924 (local address 127.0.98.66:37861)
0926 20:43:46.013961 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:46.013963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:46.014040 (+ 77us) server_negotiation.cc:407] Connection header received
0926 20:43:46.014112 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:46.014112 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:46.014120 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:46.014165 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:43:46.014307 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.014749 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.015902 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.016081 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.016131 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:46.016319 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:43:46.016342 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:43:46.016377 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:46.016468 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 20:43:46.016480 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:43:46.021196 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:46.018874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:53251 (local address 127.0.98.66:37861)
0926 20:43:46.018909 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:46.018910 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:46.019033 (+ 123us) server_negotiation.cc:407] Connection header received
0926 20:43:46.019092 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:46.019093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:46.019102 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:46.019160 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:43:46.019276 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.019703 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.020460 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.020652 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.020734 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:46.020923 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:43:46.020950 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:43:46.021017 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:46.021075 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 20:43:46.021110 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":17}
I20260926 20:43:46.021255 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:46.018814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37861 (local address 127.0.98.66:53251)
0926 20:43:46.018971 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:46.018979 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:46.019047 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:46.019163 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:46.019164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:46.019165 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:43:46.019219 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:46.019220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.019804 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.019805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:46.020379 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:46.020381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.020755 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.020756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:46.020807 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:46.020825 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:43:46.021017 (+ 192us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:43:46.021019 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:43:46.021071 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 20:43:46.021125 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260926 20:43:46.024215 2279 raft_consensus.cc:1261] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 1 FOLLOWER]: Refusing update from remote peer 5cfe9af3f0c3451789f72baa48468cad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:46.024223 2559 raft_consensus.cc:1261] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e [term 1 FOLLOWER]: Refusing update from remote peer 5cfe9af3f0c3451789f72baa48468cad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:46.024426 2638 consensus_queue.cc:1035] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:43:46.024528 2660 consensus_queue.cc:1035] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:43:46.029088 2655 mvcc.cc:205] Tried to move back new op lower bound from 7333705424993964032 to 7333705423636377600. Current Snapshot: MvccSnapshot[applied={T|T < 7333705424993964032}]
I20260926 20:43:46.036178 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:43:46.036293 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:43:46.037076 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a: Sending DeleteTablet for 3 replicas of tablet e5ca7716c20c4e93bf89cc8c588443cb
I20260926 20:43:46.037436 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ca7716c20c4e93bf89cc8c588443cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:46 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:56142
I20260926 20:43:46.037561 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 20:43:46.037663 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ca7716c20c4e93bf89cc8c588443cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:46 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:42946
W20260926 20:43:46.037710 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:43:46.037858 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ca7716c20c4e93bf89cc8c588443cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:46 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:58912
I20260926 20:43:46.038089 2671 tablet_replica.cc:330] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad: stopping tablet replica
I20260926 20:43:46.038193 2671 raft_consensus.cc:2229] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:43:46.038295 2671 raft_consensus.cc:2258] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:46.038842 2671 ts_tablet_manager.cc:1905] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:43:46.038934 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 4f2328cc27ed42488d049a6724e61232 (DEFAULT_TABLE table=test-table [id=b514f3b4f36647c681743ffac54c9c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:46.038931 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 4f2328cc27ed42488d049a6724e61232 (DEFAULT_TABLE table=test-table [id=b514f3b4f36647c681743ffac54c9c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:46.039032 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f2328cc27ed42488d049a6724e61232. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:46.039057 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f2328cc27ed42488d049a6724e61232. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:46.040376 2629 tablet_bootstrap.cc:492] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Bootstrap starting.
I20260926 20:43:46.040444 2628 tablet_bootstrap.cc:492] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e: Bootstrap starting.
I20260926 20:43:46.040690 2671 ts_tablet_manager.cc:1918] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:43:46.040753 2671 log.cc:1196] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/e5ca7716c20c4e93bf89cc8c588443cb
I20260926 20:43:46.040771 2629 tablet_bootstrap.cc:654] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:46.040827 2628 tablet_bootstrap.cc:654] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:46.041033 2671 ts_tablet_manager.cc:1939] T e5ca7716c20c4e93bf89cc8c588443cb P 5cfe9af3f0c3451789f72baa48468cad: Deleting consensus metadata
I20260926 20:43:46.041433 2107 catalog_manager.cc:4841] TS 5cfe9af3f0c3451789f72baa48468cad (127.0.98.66:37861): tablet e5ca7716c20c4e93bf89cc8c588443cb (table test-table [id=b977dc35076748939dfec4276d4bf747]) successfully deleted
I20260926 20:43:46.041445 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 4f2328cc27ed42488d049a6724e61232 (DEFAULT_TABLE table=test-table [id=b514f3b4f36647c681743ffac54c9c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:46.041540 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f2328cc27ed42488d049a6724e61232. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:46.041594 2669 tablet_replica.cc:330] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e: stopping tablet replica
I20260926 20:43:46.041702 2669 raft_consensus.cc:2229] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:43:46.041769 2669 raft_consensus.cc:2258] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:46.042233 2629 tablet_bootstrap.cc:492] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8: No bootstrap required, opened a new log
I20260926 20:43:46.042284 2629 ts_tablet_manager.cc:1397] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:43:46.042410 2669 ts_tablet_manager.cc:1905] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:43:46.042418 2629 raft_consensus.cc:348] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:46.042500 2629 raft_consensus.cc:374] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:46.042521 2629 raft_consensus.cc:729] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d3b545cc7fe4139ba6b5f9f8938a0e8, State: Initialized, Role: FOLLOWER
I20260926 20:43:46.042555 2629 consensus_queue.cc:260] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:46.042649 2629 ts_tablet_manager.cc:1428] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:46.042932 2632 tablet_bootstrap.cc:492] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad: Bootstrap starting.
I20260926 20:43:46.043291 2632 tablet_bootstrap.cc:654] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:46.043941 2632 tablet_bootstrap.cc:492] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad: No bootstrap required, opened a new log
I20260926 20:43:46.043988 2632 ts_tablet_manager.cc:1397] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:43:46.044134 2632 raft_consensus.cc:348] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:46.044195 2632 raft_consensus.cc:374] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:46.044215 2632 raft_consensus.cc:729] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cfe9af3f0c3451789f72baa48468cad, State: Initialized, Role: FOLLOWER
I20260926 20:43:46.044255 2632 consensus_queue.cc:260] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:46.044296 2669 ts_tablet_manager.cc:1918] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:43:46.044334 2632 ts_tablet_manager.cc:1428] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:46.044364 2669 log.cc:1196] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/e5ca7716c20c4e93bf89cc8c588443cb
I20260926 20:43:46.044654 2669 ts_tablet_manager.cc:1939] T e5ca7716c20c4e93bf89cc8c588443cb P d3bd7e5e06c147d88277f2c9364a0e5e: Deleting consensus metadata
I20260926 20:43:46.045059 2109 catalog_manager.cc:4841] TS d3bd7e5e06c147d88277f2c9364a0e5e (127.0.98.67:43661): tablet e5ca7716c20c4e93bf89cc8c588443cb (table test-table [id=b977dc35076748939dfec4276d4bf747]) successfully deleted
I20260926 20:43:46.047832 2670 tablet_replica.cc:330] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8: stopping tablet replica
I20260926 20:43:46.047966 2670 raft_consensus.cc:2229] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:43:46.048036 2670 raft_consensus.cc:2258] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:46.048141 2628 tablet_bootstrap.cc:492] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e: No bootstrap required, opened a new log
I20260926 20:43:46.048198 2628 ts_tablet_manager.cc:1397] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260926 20:43:46.048352 2628 raft_consensus.cc:348] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:46.048437 2628 raft_consensus.cc:374] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:46.048465 2628 raft_consensus.cc:729] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3bd7e5e06c147d88277f2c9364a0e5e, State: Initialized, Role: FOLLOWER
I20260926 20:43:46.048473 2670 ts_tablet_manager.cc:1905] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:43:46.048507 2628 consensus_queue.cc:260] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:46.048612 2628 ts_tablet_manager.cc:1428] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 20:43:46.050087 2670 ts_tablet_manager.cc:1918] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:43:46.050196 2670 log.cc:1196] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/wal/wals/e5ca7716c20c4e93bf89cc8c588443cb
I20260926 20:43:46.050447 2670 ts_tablet_manager.cc:1939] T e5ca7716c20c4e93bf89cc8c588443cb P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Deleting consensus metadata
I20260926 20:43:46.050922 2108 catalog_manager.cc:4841] TS 9d3b545cc7fe4139ba6b5f9f8938a0e8 (127.0.98.65:43077): tablet e5ca7716c20c4e93bf89cc8c588443cb (table test-table [id=b977dc35076748939dfec4276d4bf747]) successfully deleted
I20260926 20:43:46.063203 2635 raft_consensus.cc:482] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:43:46.063323 2635 raft_consensus.cc:504] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:46.063706 2635 leader_election.cc:288] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cfe9af3f0c3451789f72baa48468cad (127.0.98.66:37861), d3bd7e5e06c147d88277f2c9364a0e5e (127.0.98.67:43661)
I20260926 20:43:46.067209 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:46.063887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43661 (local address 127.0.98.65:39225)
0926 20:43:46.064135 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:46.064145 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:46.064220 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:46.064392 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:46.064393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:46.064394 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:46.064479 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:46.064481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.065366 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.065367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:46.066367 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:46.066368 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.066907 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.066908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:46.066988 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:46.066992 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:43:46.067070 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:43:46.067115 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":65,"threads_started":1}
I20260926 20:43:46.067195 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:46.063980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:39225 (local address 127.0.98.67:43661)
0926 20:43:46.064070 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:46.064072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:46.064196 (+ 124us) server_negotiation.cc:407] Connection header received
0926 20:43:46.064291 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:46.064292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:46.064301 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:46.064350 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:46.064560 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.065278 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.066444 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.066808 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.066904 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:46.066915 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:46.067078 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 20:43:46.067122 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:43:46.067241 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:46.064011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:43829 (local address 127.0.98.66:37861)
0926 20:43:46.064079 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:46.064081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:46.064130 (+ 49us) server_negotiation.cc:407] Connection header received
0926 20:43:46.064206 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:46.064207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:46.064217 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:46.064289 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:46.064457 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.064952 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.066303 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.066821 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.066893 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:46.066906 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:46.067106 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 20:43:46.067157 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:43:46.067482 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:46.063867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37861 (local address 127.0.98.65:43829)
0926 20:43:46.064061 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:46.064073 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:46.064141 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:46.064307 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:46.064307 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:46.064309 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:46.064384 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:46.064385 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.065053 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.065054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:46.066202 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:46.066205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.066941 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.066942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:46.067020 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:46.067024 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:43:46.067363 (+ 339us) client_negotiation.cc:240] Negotiation successful
0926 20:43:46.067407 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":61,"threads_started":1}
I20260926 20:43:46.067834 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2328cc27ed42488d049a6724e61232" candidate_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" is_pre_election: true
I20260926 20:43:46.067938 2559 raft_consensus.cc:2454] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d3b545cc7fe4139ba6b5f9f8938a0e8 in term 0.
I20260926 20:43:46.067924 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2328cc27ed42488d049a6724e61232" candidate_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cfe9af3f0c3451789f72baa48468cad" is_pre_election: true
I20260926 20:43:46.068056 2419 raft_consensus.cc:2454] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d3b545cc7fe4139ba6b5f9f8938a0e8 in term 0.
I20260926 20:43:46.068128 2213 leader_election.cc:302] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d3b545cc7fe4139ba6b5f9f8938a0e8, d3bd7e5e06c147d88277f2c9364a0e5e; no voters:
I20260926 20:43:46.068260 2635 raft_consensus.cc:2781] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:43:46.068310 2635 raft_consensus.cc:482] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:43:46.068339 2635 raft_consensus.cc:3037] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:46.069232 2635 raft_consensus.cc:504] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:46.069378 2635 leader_election.cc:288] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [CANDIDATE]: Term 1 election: Requested vote from peers 5cfe9af3f0c3451789f72baa48468cad (127.0.98.66:37861), d3bd7e5e06c147d88277f2c9364a0e5e (127.0.98.67:43661)
I20260926 20:43:46.069581 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2328cc27ed42488d049a6724e61232" candidate_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e"
I20260926 20:43:46.069650 2559 raft_consensus.cc:3037] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:46.069655 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2328cc27ed42488d049a6724e61232" candidate_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cfe9af3f0c3451789f72baa48468cad"
I20260926 20:43:46.069741 2419 raft_consensus.cc:3037] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:46.070382 2559 raft_consensus.cc:2454] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d3b545cc7fe4139ba6b5f9f8938a0e8 in term 1.
I20260926 20:43:46.070473 2419 raft_consensus.cc:2454] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d3b545cc7fe4139ba6b5f9f8938a0e8 in term 1.
I20260926 20:43:46.070619 2213 leader_election.cc:302] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d3b545cc7fe4139ba6b5f9f8938a0e8, d3bd7e5e06c147d88277f2c9364a0e5e; no voters:
I20260926 20:43:46.070720 2635 raft_consensus.cc:2781] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:46.070854 2635 raft_consensus.cc:686] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 1 LEADER]: Becoming Leader. State: Replica: 9d3b545cc7fe4139ba6b5f9f8938a0e8, State: Running, Role: LEADER
I20260926 20:43:46.070922 2635 consensus_queue.cc:237] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:46.071749 2122 catalog_manager.cc:5495] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d3b545cc7fe4139ba6b5f9f8938a0e8 (127.0.98.65). New cstate: current_term: 1 leader_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } health_report { overall_health: UNKNOWN } } }
I20260926 20:43:46.083892 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:46.081141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.0.98.65:43077)
0926 20:43:46.081191 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:46.081193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:46.081247 (+ 54us) server_negotiation.cc:407] Connection header received
0926 20:43:46.081330 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:46.081331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:46.081345 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:46.081402 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:43:46.081535 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.082008 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.083036 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.083297 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.083364 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:46.083521 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:43:46.083663 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:43:46.083761 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:46.083819 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 20:43:46.083840 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 20:43:46.089190 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:46.086496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37861 (local address 127.0.98.65:56577)
0926 20:43:46.086626 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:46.086636 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:46.086697 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:46.086814 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:46.086815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:46.086816 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:43:46.086864 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:46.086865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.087653 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.087654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:46.088380 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:46.088382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.088835 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.088836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:46.088878 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:46.088889 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:43:46.089047 (+ 158us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:43:46.089048 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:43:46.089078 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 20:43:46.089110 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":69,"threads_started":1}
I20260926 20:43:46.089184 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:46.086525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:56577 (local address 127.0.98.66:37861)
0926 20:43:46.086567 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:46.086569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:46.086695 (+ 126us) server_negotiation.cc:407] Connection header received
0926 20:43:46.086740 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:46.086741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:46.086750 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:46.086813 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:43:46.086905 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.087581 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.088457 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.088749 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.088818 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:46.088960 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:43:46.088993 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:43:46.089040 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:46.089088 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 20:43:46.089105 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:43:46.091495 2559 raft_consensus.cc:1261] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e [term 1 FOLLOWER]: Refusing update from remote peer 9d3b545cc7fe4139ba6b5f9f8938a0e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:46.091537 2419 raft_consensus.cc:1261] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad [term 1 FOLLOWER]: Refusing update from remote peer 9d3b545cc7fe4139ba6b5f9f8938a0e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:46.091805 2635 consensus_queue.cc:1035] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:43:46.092012 2677 consensus_queue.cc:1035] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:43:46.099884 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:46.097173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:58313 (local address 127.0.98.65:43077)
0926 20:43:46.097211 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:46.097213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:46.097259 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:43:46.097330 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:46.097331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:46.097341 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:46.097386 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:43:46.097547 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.098261 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.099086 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.099339 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.099426 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:46.099573 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:43:46.099604 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:43:46.099650 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:46.099793 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 20:43:46.099810 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:43:46.099879 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:46.097137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43077 (local address 127.0.98.66:58313)
0926 20:43:46.097217 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:46.097228 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:46.097274 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:46.097426 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:46.097426 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:46.097428 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:43:46.097468 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:46.097470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.098340 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.098341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:46.099023 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:46.099026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:46.099426 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:46.099427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:46.099481 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:46.099486 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:43:46.099649 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:43:46.099650 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:43:46.099700 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 20:43:46.099798 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 20:43:46.100189 2654 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 20:43:47.622783 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:43:47.622910 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:43:47.623781 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a: Sending DeleteTablet for 3 replicas of tablet 4f2328cc27ed42488d049a6724e61232
I20260926 20:43:47.624006 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f2328cc27ed42488d049a6724e61232 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:47 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:42946
I20260926 20:43:47.624061 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f2328cc27ed42488d049a6724e61232 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:47 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:56142
I20260926 20:43:47.624190 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f2328cc27ed42488d049a6724e61232 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:47 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:58912
I20260926 20:43:47.624454 2708 tablet_replica.cc:330] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad: stopping tablet replica
I20260926 20:43:47.624521 2708 raft_consensus.cc:2229] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:43:47.624562 2708 raft_consensus.cc:2258] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:47.624709 2707 tablet_replica.cc:330] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e: stopping tablet replica
I20260926 20:43:47.624796 2708 ts_tablet_manager.cc:1905] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:43:47.624825 2707 raft_consensus.cc:2229] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:43:47.624861 2707 raft_consensus.cc:2258] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:47.626753 2708 ts_tablet_manager.cc:1918] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:43:47.626819 2708 log.cc:1196] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/4f2328cc27ed42488d049a6724e61232
I20260926 20:43:47.626825 2707 ts_tablet_manager.cc:1905] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:43:47.624200 2706 tablet_replica.cc:330] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8: stopping tablet replica
I20260926 20:43:47.626974 2706 raft_consensus.cc:2229] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:43:47.627065 2708 ts_tablet_manager.cc:1939] T 4f2328cc27ed42488d049a6724e61232 P 5cfe9af3f0c3451789f72baa48468cad: Deleting consensus metadata
I20260926 20:43:47.627072 2706 raft_consensus.cc:2258] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:47.627395 2107 catalog_manager.cc:4841] TS 5cfe9af3f0c3451789f72baa48468cad (127.0.98.66:37861): tablet 4f2328cc27ed42488d049a6724e61232 (table test-table [id=b514f3b4f36647c681743ffac54c9c49]) successfully deleted
I20260926 20:43:47.627957 2706 ts_tablet_manager.cc:1905] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:43:47.628921 2707 ts_tablet_manager.cc:1918] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:43:47.628983 2707 log.cc:1196] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/4f2328cc27ed42488d049a6724e61232
I20260926 20:43:47.629212 2707 ts_tablet_manager.cc:1939] T 4f2328cc27ed42488d049a6724e61232 P d3bd7e5e06c147d88277f2c9364a0e5e: Deleting consensus metadata
I20260926 20:43:47.629518 2706 ts_tablet_manager.cc:1918] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:43:47.629591 2109 catalog_manager.cc:4841] TS d3bd7e5e06c147d88277f2c9364a0e5e (127.0.98.67:43661): tablet 4f2328cc27ed42488d049a6724e61232 (table test-table [id=b514f3b4f36647c681743ffac54c9c49]) successfully deleted
I20260926 20:43:47.629626 2706 log.cc:1196] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/wal/wals/4f2328cc27ed42488d049a6724e61232
I20260926 20:43:47.629844 2706 ts_tablet_manager.cc:1939] T 4f2328cc27ed42488d049a6724e61232 P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Deleting consensus metadata
I20260926 20:43:47.630129 2108 catalog_manager.cc:4841] TS 9d3b545cc7fe4139ba6b5f9f8938a0e8 (127.0.98.65:43077): tablet 4f2328cc27ed42488d049a6724e61232 (table test-table [id=b514f3b4f36647c681743ffac54c9c49]) successfully deleted
Sep 26 20:43:47 dist-test-slave-4hd3 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455427, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:43:47 dist-test-slave-4hd3 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455427, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:47.647495 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:47.640220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39830 (local address 127.0.98.126:38439)
0926 20:43:47.640367 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:47.640369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:47.640391 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:43:47.640448 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:47.640449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:47.640458 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:47.640516 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:47.641081 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.641700 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.642460 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.642685 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.642739 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:47.645034 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:47.645036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:47.645036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:47.645050 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:47.646338 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:47.646709 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:47.646710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:47.646711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:47.646770 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:47.646963 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:47.646964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:47.646964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:47.647085 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:47.647194 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:47.647301 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 20:43:47.647380 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":99,"threads_started":1}
I20260926 20:43:47.649199 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:43:47.649374 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:43:47.650724 2259 tablet_service.cc:1467] Processing CreateTablet for tablet c0c14d63c58442d0b486190306cf7b5e (DEFAULT_TABLE table=test-table [id=ce493c8f21c946768adc89c771560599]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:47.650748 2539 tablet_service.cc:1467] Processing CreateTablet for tablet c0c14d63c58442d0b486190306cf7b5e (DEFAULT_TABLE table=test-table [id=ce493c8f21c946768adc89c771560599]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:47.650840 2398 tablet_service.cc:1467] Processing CreateTablet for tablet c0c14d63c58442d0b486190306cf7b5e (DEFAULT_TABLE table=test-table [id=ce493c8f21c946768adc89c771560599]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:47.650874 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c14d63c58442d0b486190306cf7b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:47.650889 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c14d63c58442d0b486190306cf7b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:47.650946 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c14d63c58442d0b486190306cf7b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:47.652360 2718 tablet_bootstrap.cc:492] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Bootstrap starting.
I20260926 20:43:47.652560 2717 tablet_bootstrap.cc:492] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e: Bootstrap starting.
I20260926 20:43:47.652649 2719 tablet_bootstrap.cc:492] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad: Bootstrap starting.
I20260926 20:43:47.652803 2718 tablet_bootstrap.cc:654] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:47.653004 2719 tablet_bootstrap.cc:654] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:47.653004 2717 tablet_bootstrap.cc:654] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:47.653647 2717 tablet_bootstrap.cc:492] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e: No bootstrap required, opened a new log
I20260926 20:43:47.653707 2717 ts_tablet_manager.cc:1397] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:43:47.653858 2719 tablet_bootstrap.cc:492] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad: No bootstrap required, opened a new log
I20260926 20:43:47.653841 2717 raft_consensus.cc:348] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:47.653903 2719 ts_tablet_manager.cc:1397] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:43:47.653914 2717 raft_consensus.cc:374] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:47.653939 2717 raft_consensus.cc:729] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3bd7e5e06c147d88277f2c9364a0e5e, State: Initialized, Role: FOLLOWER
I20260926 20:43:47.653992 2717 consensus_queue.cc:260] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:47.654011 2719 raft_consensus.cc:348] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:47.654074 2719 raft_consensus.cc:374] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:47.654100 2719 raft_consensus.cc:729] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cfe9af3f0c3451789f72baa48468cad, State: Initialized, Role: FOLLOWER
I20260926 20:43:47.654162 2717 ts_tablet_manager.cc:1428] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:43:47.654184 2719 consensus_queue.cc:260] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:47.654287 2719 ts_tablet_manager.cc:1428] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:47.654397 2718 tablet_bootstrap.cc:492] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8: No bootstrap required, opened a new log
I20260926 20:43:47.654449 2718 ts_tablet_manager.cc:1397] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:43:47.654565 2718 raft_consensus.cc:348] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:47.654637 2718 raft_consensus.cc:374] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:47.654661 2718 raft_consensus.cc:729] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d3b545cc7fe4139ba6b5f9f8938a0e8, State: Initialized, Role: FOLLOWER
I20260926 20:43:47.654803 2718 consensus_queue.cc:260] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:47.654927 2718 ts_tablet_manager.cc:1428] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:47.672139 2723 raft_consensus.cc:482] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:43:47.672217 2723 raft_consensus.cc:504] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:47.672577 2723 leader_election.cc:288] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d3b545cc7fe4139ba6b5f9f8938a0e8 (127.0.98.65:43077), 5cfe9af3f0c3451789f72baa48468cad (127.0.98.66:37861)
I20260926 20:43:47.675617 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:47.672773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37861 (local address 127.0.98.67:56353)
0926 20:43:47.672931 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:47.672942 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:47.672994 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:47.673209 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:47.673210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:47.673211 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:47.673274 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:47.673275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.673988 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.673989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:47.674792 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:47.674794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.675334 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.675334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:47.675407 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:47.675412 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:43:47.675466 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:43:47.675496 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260926 20:43:47.675632 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:47.672845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:56353 (local address 127.0.98.66:37861)
0926 20:43:47.672981 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:47.672983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:47.672991 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:43:47.673088 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:47.673089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:47.673101 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:47.673165 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:47.673356 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.673905 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.674876 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.675256 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.675310 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:47.675318 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:47.675490 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 20:43:47.675543 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260926 20:43:47.675741 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:47.672731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43077 (local address 127.0.98.67:60259)
0926 20:43:47.672900 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:47.672911 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:47.672969 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:47.673169 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:47.673170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:47.673172 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:47.673277 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:47.673278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.673935 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.673936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:47.674794 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:47.674796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.675403 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.675404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:47.675476 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:47.675481 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:43:47.675537 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 20:43:47.675584 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":43,"thread_start_us":104,"threads_started":1}
I20260926 20:43:47.675863 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c14d63c58442d0b486190306cf7b5e" candidate_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cfe9af3f0c3451789f72baa48468cad" is_pre_election: true
I20260926 20:43:47.675932 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:47.672779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:60259 (local address 127.0.98.65:43077)
0926 20:43:47.672945 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:47.672947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:47.672957 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:43:47.673051 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:47.673052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:47.673063 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:47.673156 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:47.673345 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.673855 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.674904 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.675306 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.675375 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:47.675387 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:47.675556 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 20:43:47.675620 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260926 20:43:47.675967 2419 raft_consensus.cc:2454] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3bd7e5e06c147d88277f2c9364a0e5e in term 0.
I20260926 20:43:47.676079 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c14d63c58442d0b486190306cf7b5e" candidate_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" is_pre_election: true
I20260926 20:43:47.676141 2491 leader_election.cc:302] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cfe9af3f0c3451789f72baa48468cad, d3bd7e5e06c147d88277f2c9364a0e5e; no voters:
I20260926 20:43:47.676191 2279 raft_consensus.cc:2454] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3bd7e5e06c147d88277f2c9364a0e5e in term 0.
I20260926 20:43:47.676257 2723 raft_consensus.cc:2781] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:43:47.676290 2723 raft_consensus.cc:482] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:43:47.676313 2723 raft_consensus.cc:3037] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:47.676957 2723 raft_consensus.cc:504] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:47.677088 2723 leader_election.cc:288] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [CANDIDATE]: Term 1 election: Requested vote from peers 9d3b545cc7fe4139ba6b5f9f8938a0e8 (127.0.98.65:43077), 5cfe9af3f0c3451789f72baa48468cad (127.0.98.66:37861)
I20260926 20:43:47.677218 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c14d63c58442d0b486190306cf7b5e" candidate_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8"
I20260926 20:43:47.677258 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c14d63c58442d0b486190306cf7b5e" candidate_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cfe9af3f0c3451789f72baa48468cad"
I20260926 20:43:47.677305 2279 raft_consensus.cc:3037] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:47.677314 2419 raft_consensus.cc:3037] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:47.678058 2419 raft_consensus.cc:2454] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3bd7e5e06c147d88277f2c9364a0e5e in term 1.
I20260926 20:43:47.678071 2279 raft_consensus.cc:2454] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3bd7e5e06c147d88277f2c9364a0e5e in term 1.
I20260926 20:43:47.678231 2492 leader_election.cc:302] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d3b545cc7fe4139ba6b5f9f8938a0e8, d3bd7e5e06c147d88277f2c9364a0e5e; no voters:
I20260926 20:43:47.678363 2723 raft_consensus.cc:2781] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:47.678498 2723 raft_consensus.cc:686] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [term 1 LEADER]: Becoming Leader. State: Replica: d3bd7e5e06c147d88277f2c9364a0e5e, State: Running, Role: LEADER
I20260926 20:43:47.678575 2723 consensus_queue.cc:237] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:47.679260 2121 catalog_manager.cc:5495] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e reported cstate change: term changed from 0 to 1, leader changed from <none> to d3bd7e5e06c147d88277f2c9364a0e5e (127.0.98.67). New cstate: current_term: 1 leader_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } health_report { overall_health: HEALTHY } } }
I20260926 20:43:47.693121 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:47.690872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56158 (local address 127.0.98.67:43661)
0926 20:43:47.691010 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:47.691014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:47.691026 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:43:47.691056 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:47.691056 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:47.691073 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:47.691148 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:43:47.691253 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.691706 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.692444 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.692644 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.692697 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:47.692813 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:43:47.692908 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:43:47.692958 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:47.692991 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 20:43:47.693005 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":95,"threads_started":1}
I20260926 20:43:47.697566 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:47.695558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:53645 (local address 127.0.98.66:37861)
0926 20:43:47.695602 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:47.695604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:47.695656 (+ 52us) server_negotiation.cc:407] Connection header received
0926 20:43:47.695700 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:47.695700 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:47.695739 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:47.695792 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:43:47.695887 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.696368 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.697060 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.697229 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.697288 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:47.697415 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:43:47.697438 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:43:47.697474 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:47.697504 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 20:43:47.697519 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 20:43:47.697621 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:47.695423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37861 (local address 127.0.98.67:53645)
0926 20:43:47.695575 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:47.695610 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:47.695660 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:47.695795 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:47.695796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:47.695797 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:43:47.695842 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:47.695843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.696426 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.696426 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:47.696996 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:47.696997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.697290 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.697291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:47.697337 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:47.697349 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:43:47.697473 (+ 124us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:43:47.697474 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:43:47.697504 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 20:43:47.697532 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260926 20:43:47.699568 2279 raft_consensus.cc:1261] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 1 FOLLOWER]: Refusing update from remote peer d3bd7e5e06c147d88277f2c9364a0e5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:47.699568 2419 raft_consensus.cc:1261] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad [term 1 FOLLOWER]: Refusing update from remote peer d3bd7e5e06c147d88277f2c9364a0e5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:47.699867 2723 consensus_queue.cc:1035] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:43:47.699982 2728 consensus_queue.cc:1035] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:43:47.706806 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:47.704559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:41419 (local address 127.0.98.67:43661)
0926 20:43:47.704599 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:47.704601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:47.704721 (+ 120us) server_negotiation.cc:407] Connection header received
0926 20:43:47.704772 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:47.704773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:47.704781 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:47.704817 (+ 36us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:43:47.704907 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.705372 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.706096 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.706396 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.706464 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:47.706614 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:43:47.706638 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:43:47.706675 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:47.706745 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 20:43:47.706760 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:43:47.706832 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:47.704518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43661 (local address 127.0.98.66:41419)
0926 20:43:47.704664 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:47.704674 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:47.704734 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:47.704828 (+ 94us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:47.704829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:47.704830 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:43:47.704867 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:47.704868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.705446 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.705446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:47.705997 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:47.705999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.706471 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.706472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:47.706537 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:47.706542 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:43:47.706708 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:43:47.706710 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:43:47.706744 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 20:43:47.706779 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":104,"threads_started":1}
I20260926 20:43:47.713013 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:43:47.713117 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:43:47.713692 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a: Sending DeleteTablet for 3 replicas of tablet c0c14d63c58442d0b486190306cf7b5e
I20260926 20:43:47.713939 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c14d63c58442d0b486190306cf7b5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:47 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:58912
I20260926 20:43:47.713946 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c14d63c58442d0b486190306cf7b5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:47 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:56142
I20260926 20:43:47.713959 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c14d63c58442d0b486190306cf7b5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:47 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:42946
I20260926 20:43:47.714025 2708 tablet_replica.cc:330] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad: stopping tablet replica
I20260926 20:43:47.714035 2706 tablet_replica.cc:330] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8: stopping tablet replica
I20260926 20:43:47.714037 2707 tablet_replica.cc:330] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e: stopping tablet replica
I20260926 20:43:47.714076 2708 raft_consensus.cc:2229] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:43:47.714097 2707 raft_consensus.cc:2229] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:43:47.714110 2706 raft_consensus.cc:2229] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:43:47.714114 2708 raft_consensus.cc:2258] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:47.714143 2706 raft_consensus.cc:2258] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:47.714169 2707 raft_consensus.cc:2258] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:47.714448 2707 ts_tablet_manager.cc:1905] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:43:47.714588 2706 ts_tablet_manager.cc:1905] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:43:47.714668 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:43:47.714797 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:43:47.716022 2707 ts_tablet_manager.cc:1918] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:43:47.716078 2707 log.cc:1196] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/c0c14d63c58442d0b486190306cf7b5e
I20260926 20:43:47.716223 2706 ts_tablet_manager.cc:1918] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:43:47.716282 2706 log.cc:1196] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-0/wal/wals/c0c14d63c58442d0b486190306cf7b5e
I20260926 20:43:47.716308 2707 ts_tablet_manager.cc:1939] T c0c14d63c58442d0b486190306cf7b5e P d3bd7e5e06c147d88277f2c9364a0e5e: Deleting consensus metadata
I20260926 20:43:47.716462 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa4b1ef366949f1a8bd913134c4e50a (DEFAULT_TABLE table=test-table [id=4dcd99a87adc421a9efbeb2393d208fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:47.716501 2706 ts_tablet_manager.cc:1939] T c0c14d63c58442d0b486190306cf7b5e P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Deleting consensus metadata
I20260926 20:43:47.716564 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa4b1ef366949f1a8bd913134c4e50a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:47.716830 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa4b1ef366949f1a8bd913134c4e50a (DEFAULT_TABLE table=test-table [id=4dcd99a87adc421a9efbeb2393d208fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:47.716881 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa4b1ef366949f1a8bd913134c4e50a (DEFAULT_TABLE table=test-table [id=4dcd99a87adc421a9efbeb2393d208fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:47.716924 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa4b1ef366949f1a8bd913134c4e50a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:47.716960 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa4b1ef366949f1a8bd913134c4e50a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:47.717878 2719 tablet_bootstrap.cc:492] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad: Bootstrap starting.
I20260926 20:43:47.718032 2717 tablet_bootstrap.cc:492] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e: Bootstrap starting.
I20260926 20:43:47.718073 2108 catalog_manager.cc:4841] TS 9d3b545cc7fe4139ba6b5f9f8938a0e8 (127.0.98.65:43077): tablet c0c14d63c58442d0b486190306cf7b5e (table test-table [id=ce493c8f21c946768adc89c771560599]) successfully deleted
I20260926 20:43:47.718197 2719 tablet_bootstrap.cc:654] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:47.718686 2719 tablet_bootstrap.cc:492] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad: No bootstrap required, opened a new log
I20260926 20:43:47.718729 2719 ts_tablet_manager.cc:1397] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:43:47.718834 2717 tablet_bootstrap.cc:654] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:47.718817 2708 ts_tablet_manager.cc:1905] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:43:47.719131 2718 tablet_bootstrap.cc:492] T 7aa4b1ef366949f1a8bd913134c4e50a P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Bootstrap starting.
I20260926 20:43:47.719416 2718 tablet_bootstrap.cc:654] T 7aa4b1ef366949f1a8bd913134c4e50a P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:47.719417 2717 tablet_bootstrap.cc:492] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e: No bootstrap required, opened a new log
I20260926 20:43:47.719460 2717 ts_tablet_manager.cc:1397] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:43:47.719609 2717 raft_consensus.cc:348] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:47.719668 2717 raft_consensus.cc:374] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:47.719689 2717 raft_consensus.cc:729] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3bd7e5e06c147d88277f2c9364a0e5e, State: Initialized, Role: FOLLOWER
I20260926 20:43:47.719756 2717 consensus_queue.cc:260] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:47.719875 2717 ts_tablet_manager.cc:1428] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:47.720147 2109 catalog_manager.cc:4841] TS d3bd7e5e06c147d88277f2c9364a0e5e (127.0.98.67:43661): tablet c0c14d63c58442d0b486190306cf7b5e (table test-table [id=ce493c8f21c946768adc89c771560599]) successfully deleted
I20260926 20:43:47.720340 2708 ts_tablet_manager.cc:1918] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:43:47.720396 2708 log.cc:1196] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/c0c14d63c58442d0b486190306cf7b5e
I20260926 20:43:47.720615 2708 ts_tablet_manager.cc:1939] T c0c14d63c58442d0b486190306cf7b5e P 5cfe9af3f0c3451789f72baa48468cad: Deleting consensus metadata
I20260926 20:43:47.720927 2107 catalog_manager.cc:4841] TS 5cfe9af3f0c3451789f72baa48468cad (127.0.98.66:37861): tablet c0c14d63c58442d0b486190306cf7b5e (table test-table [id=ce493c8f21c946768adc89c771560599]) successfully deleted
I20260926 20:43:47.718828 2719 raft_consensus.cc:348] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:47.720983 2719 raft_consensus.cc:374] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:47.721010 2719 raft_consensus.cc:729] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cfe9af3f0c3451789f72baa48468cad, State: Initialized, Role: FOLLOWER
I20260926 20:43:47.721060 2719 consensus_queue.cc:260] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:47.721160 2719 ts_tablet_manager.cc:1428] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 20:43:47.721577 2718 tablet_bootstrap.cc:492] T 7aa4b1ef366949f1a8bd913134c4e50a P 9d3b545cc7fe4139ba6b5f9f8938a0e8: No bootstrap required, opened a new log
I20260926 20:43:47.721628 2718 ts_tablet_manager.cc:1397] T 7aa4b1ef366949f1a8bd913134c4e50a P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 20:43:47.721732 2718 raft_consensus.cc:348] T 7aa4b1ef366949f1a8bd913134c4e50a P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:47.721807 2718 raft_consensus.cc:374] T 7aa4b1ef366949f1a8bd913134c4e50a P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:47.721833 2718 raft_consensus.cc:729] T 7aa4b1ef366949f1a8bd913134c4e50a P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d3b545cc7fe4139ba6b5f9f8938a0e8, State: Initialized, Role: FOLLOWER
I20260926 20:43:47.721873 2718 consensus_queue.cc:260] T 7aa4b1ef366949f1a8bd913134c4e50a P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:47.721967 2718 ts_tablet_manager.cc:1428] T 7aa4b1ef366949f1a8bd913134c4e50a P 9d3b545cc7fe4139ba6b5f9f8938a0e8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:47.727321 2688 raft_consensus.cc:482] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:43:47.727407 2688 raft_consensus.cc:504] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:47.727566 2688 leader_election.cc:288] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d3b545cc7fe4139ba6b5f9f8938a0e8 (127.0.98.65:43077), d3bd7e5e06c147d88277f2c9364a0e5e (127.0.98.67:43661)
I20260926 20:43:47.727752 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa4b1ef366949f1a8bd913134c4e50a" candidate_uuid: "5cfe9af3f0c3451789f72baa48468cad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" is_pre_election: true
I20260926 20:43:47.727793 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa4b1ef366949f1a8bd913134c4e50a" candidate_uuid: "5cfe9af3f0c3451789f72baa48468cad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" is_pre_election: true
I20260926 20:43:47.727843 2279 raft_consensus.cc:2454] T 7aa4b1ef366949f1a8bd913134c4e50a P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cfe9af3f0c3451789f72baa48468cad in term 0.
I20260926 20:43:47.727859 2559 raft_consensus.cc:2454] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cfe9af3f0c3451789f72baa48468cad in term 0.
I20260926 20:43:47.727970 2352 leader_election.cc:302] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cfe9af3f0c3451789f72baa48468cad, 9d3b545cc7fe4139ba6b5f9f8938a0e8; no voters:
I20260926 20:43:47.728053 2688 raft_consensus.cc:2781] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:43:47.728089 2688 raft_consensus.cc:482] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:43:47.728118 2688 raft_consensus.cc:3037] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:47.728669 2688 raft_consensus.cc:504] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:47.728791 2688 leader_election.cc:288] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [CANDIDATE]: Term 1 election: Requested vote from peers 9d3b545cc7fe4139ba6b5f9f8938a0e8 (127.0.98.65:43077), d3bd7e5e06c147d88277f2c9364a0e5e (127.0.98.67:43661)
I20260926 20:43:47.728909 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa4b1ef366949f1a8bd913134c4e50a" candidate_uuid: "5cfe9af3f0c3451789f72baa48468cad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8"
I20260926 20:43:47.728974 2279 raft_consensus.cc:3037] T 7aa4b1ef366949f1a8bd913134c4e50a P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:47.728948 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa4b1ef366949f1a8bd913134c4e50a" candidate_uuid: "5cfe9af3f0c3451789f72baa48468cad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e"
I20260926 20:43:47.729012 2559 raft_consensus.cc:3037] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:47.729592 2279 raft_consensus.cc:2454] T 7aa4b1ef366949f1a8bd913134c4e50a P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cfe9af3f0c3451789f72baa48468cad in term 1.
I20260926 20:43:47.729663 2559 raft_consensus.cc:2454] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cfe9af3f0c3451789f72baa48468cad in term 1.
I20260926 20:43:47.729737 2352 leader_election.cc:302] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cfe9af3f0c3451789f72baa48468cad, 9d3b545cc7fe4139ba6b5f9f8938a0e8; no voters:
I20260926 20:43:47.729820 2688 raft_consensus.cc:2781] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:47.729869 2688 raft_consensus.cc:686] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [term 1 LEADER]: Becoming Leader. State: Replica: 5cfe9af3f0c3451789f72baa48468cad, State: Running, Role: LEADER
I20260926 20:43:47.729914 2688 consensus_queue.cc:237] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } }
I20260926 20:43:47.730371 2121 catalog_manager.cc:5495] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cfe9af3f0c3451789f72baa48468cad (127.0.98.66). New cstate: current_term: 1 leader_uuid: "5cfe9af3f0c3451789f72baa48468cad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cfe9af3f0c3451789f72baa48468cad" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 } health_report { overall_health: UNKNOWN } } }
I20260926 20:43:47.738250 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:47.735730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58936 (local address 127.0.98.66:37861)
0926 20:43:47.735771 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:47.735773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:47.735824 (+ 51us) server_negotiation.cc:407] Connection header received
0926 20:43:47.735895 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:47.735895 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:47.735904 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:47.735950 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:43:47.736060 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.736525 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.737601 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:47.737805 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:47.737861 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:47.738031 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:43:47.738056 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:43:47.738094 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:47.738179 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 20:43:47.738196 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:43:47.740624 2559 raft_consensus.cc:1261] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e [term 1 FOLLOWER]: Refusing update from remote peer 5cfe9af3f0c3451789f72baa48468cad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:47.740677 2279 raft_consensus.cc:1261] T 7aa4b1ef366949f1a8bd913134c4e50a P 9d3b545cc7fe4139ba6b5f9f8938a0e8 [term 1 FOLLOWER]: Refusing update from remote peer 5cfe9af3f0c3451789f72baa48468cad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:47.740845 2636 consensus_queue.cc:1035] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3bd7e5e06c147d88277f2c9364a0e5e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:43:47.740949 2693 consensus_queue.cc:1035] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d3b545cc7fe4139ba6b5f9f8938a0e8" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 20:43:47.745558 2686 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 20:43:49.265035 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:43:49.265146 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:43:49.266115 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P adf28b02cb3348d1b8646f17c2bb025a: Sending DeleteTablet for 3 replicas of tablet 7aa4b1ef366949f1a8bd913134c4e50a
I20260926 20:43:49.266386 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa4b1ef366949f1a8bd913134c4e50a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:49 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:42946
I20260926 20:43:49.266440 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa4b1ef366949f1a8bd913134c4e50a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:49 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:56142
I20260926 20:43:49.266566 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 2194
I20260926 20:43:49.267047 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa4b1ef366949f1a8bd913134c4e50a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:49 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:58912
I20260926 20:43:49.268075 2770 tablet_replica.cc:330] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad: stopping tablet replica
I20260926 20:43:49.268147 2770 raft_consensus.cc:2229] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:43:49.268195 2770 raft_consensus.cc:2258] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:49.268438 2770 ts_tablet_manager.cc:1905] T 7aa4b1ef366949f1a8bd913134c4e50a P 5cfe9af3f0c3451789f72baa48468cad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:43:49.269196 2769 tablet_replica.cc:330] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e: stopping tablet replica
I20260926 20:43:49.269254 2769 raft_consensus.cc:2229] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:43:49.269318 2769 raft_consensus.cc:2258] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:49.269542 2769 ts_tablet_manager.cc:1905] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:43:49.273749 2108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:43077 (error 108)
I20260926 20:43:49.273996 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 2334
I20260926 20:43:49.275457 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:49.274153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43077 (local address 127.0.0.1:42962)
0926 20:43:49.275358 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:49.275414 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:43077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1179,"thread_start_us":59,"threads_started":1}
W20260926 20:43:49.275765 2108 catalog_manager.cc:4584] TS 9d3b545cc7fe4139ba6b5f9f8938a0e8 (127.0.98.65:43077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7aa4b1ef366949f1a8bd913134c4e50a: Network error: Client connection negotiation failed: client connection to 127.0.98.65:43077: connect: Connection refused (error 111)
I20260926 20:43:49.277956 2769 ts_tablet_manager.cc:1918] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:43:49.278141 2769 log.cc:1196] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/7aa4b1ef366949f1a8bd913134c4e50a
I20260926 20:43:49.278399 2769 ts_tablet_manager.cc:1939] T 7aa4b1ef366949f1a8bd913134c4e50a P d3bd7e5e06c147d88277f2c9364a0e5e: Deleting consensus metadata
I20260926 20:43:49.282130 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 2474
I20260926 20:43:49.282163 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:49.282028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37861 (local address 127.0.0.1:58942)
0926 20:43:49.282085 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:49.282123 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.66:37861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 20:43:49.300323 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:49.300177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43661 (local address 127.0.0.1:56166)
0926 20:43:49.300239 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:49.300271 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.67:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 20:43:49.300309 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 2090
2026-09-26T20:43:49Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5845 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:43:49 dist-test-slave-4hd3 krb5kdc[2775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:43:49 dist-test-slave-4hd3 krb5kdc[2775](info): set up 2 sockets
Sep 26 20:43:49 dist-test-slave-4hd3 krb5kdc[2775](info): commencing operation
krb5kdc: starting...
W20260926 20:43:50.423483 393 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:43:50 dist-test-slave-4hd3 krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:43:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:43:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:43:50.540800 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33721
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:33721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:50.621534 2788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:50.621794 2788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:50.621833 2788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:50.622992 2788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:43:50.623060 2788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:50.623083 2788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:50.623104 2788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:43:50.623133 2788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:43:50.624619 2788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:33721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:50.624895 2788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:50.625114 2788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:50.630155 2794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:50.630191 2788 server_base.cc:1034] running on GCE node
W20260926 20:43:50.630177 2793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:50.630169 2796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:50.630772 2788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:50.631027 2788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:50.632179 2788 hybrid_clock.cc:648] HybridClock initialized: now 1790455430632152 us; error 42 us; skew 500 ppm
Sep 26 20:43:50 dist-test-slave-4hd3 krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:50.634807 2788 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:43:50.635275 2788 webserver.cc:460] Webserver started at http://127.0.98.126:38669/ using document root <none> and password file <none>
I20260926 20:43:50.635490 2788 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:50.635556 2788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:50.635758 2788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:50.636770 2788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "7e333da33cb2484383878eb27e322519"
format_stamp: "Formatted at 2026-09-26 20:43:50 on dist-test-slave-4hd3"
server_key: "d6adc4dea6affb1d3a31700d23e540e1"
server_key_iv: "3fbcc0d484d739013b445e815b85f95d"
server_key_version: "encryptionkey@0"
I20260926 20:43:50.637143 2788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "7e333da33cb2484383878eb27e322519"
format_stamp: "Formatted at 2026-09-26 20:43:50 on dist-test-slave-4hd3"
server_key: "d6adc4dea6affb1d3a31700d23e540e1"
server_key_iv: "3fbcc0d484d739013b445e815b85f95d"
server_key_version: "encryptionkey@0"
I20260926 20:43:50.638420 2788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:43:50.639158 2803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:50.639339 2788 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:43:50.639437 2788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "7e333da33cb2484383878eb27e322519"
format_stamp: "Formatted at 2026-09-26 20:43:50 on dist-test-slave-4hd3"
server_key: "d6adc4dea6affb1d3a31700d23e540e1"
server_key_iv: "3fbcc0d484d739013b445e815b85f95d"
server_key_version: "encryptionkey@0"
I20260926 20:43:50.639513 2788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:50.662072 2788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:50.664517 2788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:50.664686 2788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:50.668560 2788 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:33721
I20260926 20:43:50.668895 2855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:33721 every 8 connection(s)
I20260926 20:43:50.668933 2788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:43:50.669620 2856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:50.671969 2856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519: Bootstrap starting.
I20260926 20:43:50.672546 2856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:50.672839 2856 log.cc:824] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:50.673511 2856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519: No bootstrap required, opened a new log
I20260926 20:43:50.675060 2856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e333da33cb2484383878eb27e322519" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33721 } }
I20260926 20:43:50.675215 2856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:50.675238 2856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e333da33cb2484383878eb27e322519, State: Initialized, Role: FOLLOWER
I20260926 20:43:50.675407 2856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e333da33cb2484383878eb27e322519" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33721 } }
I20260926 20:43:50.675508 2856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:43:50.675544 2856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:43:50.675536 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 2788
I20260926 20:43:50.675588 2856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:50.675632 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:43:50.675848 393 external_mini_cluster.cc:1428] Setting key fc87eef48c85d137101b5a2709cf6acb
I20260926 20:43:50.676748 2856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e333da33cb2484383878eb27e322519" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33721 } }
I20260926 20:43:50.676879 2856 leader_election.cc:302] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e333da33cb2484383878eb27e322519; no voters:
I20260926 20:43:50.677057 2856 leader_election.cc:288] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:43:50.677119 2861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:50.677251 2861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519 [term 1 LEADER]: Becoming Leader. State: Replica: 7e333da33cb2484383878eb27e322519, State: Running, Role: LEADER
I20260926 20:43:50.677331 2856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:43:50.677377 2861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e333da33cb2484383878eb27e322519" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33721 } }
I20260926 20:43:50.678342 2862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e333da33cb2484383878eb27e322519. Latest consensus state: current_term: 1 leader_uuid: "7e333da33cb2484383878eb27e322519" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e333da33cb2484383878eb27e322519" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33721 } } }
I20260926 20:43:50.678424 2862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519 [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:50.678588 2863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e333da33cb2484383878eb27e322519" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e333da33cb2484383878eb27e322519" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33721 } } }
I20260926 20:43:50.678697 2863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519 [sys.catalog]: This master's current role is: LEADER
W20260926 20:43:50.679188 2877 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:43:50.679266 2877 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:43:50.679345 2874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:43:50.679494 2874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:43:50.681053 2874 catalog_manager.cc:1347] Generated new cluster ID: 393c098401f44814b16020475747de41
I20260926 20:43:50.681114 2874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:43:50 dist-test-slave-4hd3 krb5kdc[2775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:50.687628 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:50.676342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59730 (local address 127.0.98.126:33721)
0926 20:43:50.676514 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:50.676523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:50.676558 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:43:50.676651 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:50.676658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:50.676700 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:50.676807 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:50.677590 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:50.679052 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:50.680118 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:50.680728 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:50.680834 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:50.683518 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:50.683522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:50.683523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:50.683536 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:50.685997 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:50.686381 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:50.686385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:50.686385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:50.686507 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:50.686716 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:50.686717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:50.686717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:50.687028 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:50.687216 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:50.687287 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 20:43:50.687430 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":101,"threads_started":1}
I20260926 20:43:50.689488 2874 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:43:50.689987 2874 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:43:50.694064 2874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e333da33cb2484383878eb27e322519: Generated new TSK 0
I20260926 20:43:50.694243 2874 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260926 20:43:50.722384 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33721
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:50.802788 2882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:50.803020 2882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:50.803057 2882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:50.804220 2882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:50.804289 2882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:50.804337 2882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:43:50.805842 2882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:50.806108 2882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:50.806342 2882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:50.811563 2890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:50.811520 2887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:50.811550 2888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:50.811882 2882 server_base.cc:1034] running on GCE node
I20260926 20:43:50.812096 2882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:50.812336 2882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:50.813481 2882 hybrid_clock.cc:648] HybridClock initialized: now 1790455430813461 us; error 35 us; skew 500 ppm
Sep 26 20:43:50 dist-test-slave-4hd3 krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:50.816206 2882 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:43:50.816637 2882 webserver.cc:460] Webserver started at http://127.0.98.65:42701/ using document root <none> and password file <none>
I20260926 20:43:50.816888 2882 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:50.816960 2882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:50.817111 2882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:50.818143 2882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "dda976e1faf04fb1b8d52aafe7d57cfb"
format_stamp: "Formatted at 2026-09-26 20:43:50 on dist-test-slave-4hd3"
server_key: "5cbc0bc0a01311f9f55d3163903bddfe"
server_key_iv: "354cd224779cbcb101b5a4bae1a16b19"
server_key_version: "encryptionkey@0"
I20260926 20:43:50.818508 2882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "dda976e1faf04fb1b8d52aafe7d57cfb"
format_stamp: "Formatted at 2026-09-26 20:43:50 on dist-test-slave-4hd3"
server_key: "5cbc0bc0a01311f9f55d3163903bddfe"
server_key_iv: "354cd224779cbcb101b5a4bae1a16b19"
server_key_version: "encryptionkey@0"
I20260926 20:43:50.819839 2882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:43:50.820564 2897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:50.820737 2882 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:43:50.820834 2882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "dda976e1faf04fb1b8d52aafe7d57cfb"
format_stamp: "Formatted at 2026-09-26 20:43:50 on dist-test-slave-4hd3"
server_key: "5cbc0bc0a01311f9f55d3163903bddfe"
server_key_iv: "354cd224779cbcb101b5a4bae1a16b19"
server_key_version: "encryptionkey@0"
I20260926 20:43:50.820919 2882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:50.831087 2882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:50.832970 2882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:50.833127 2882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:50.833359 2882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:50.833643 2882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:50.833696 2882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:50.833735 2882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:50.833763 2882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:50.839468 2882 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:40491
I20260926 20:43:50.839540 3010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:40491 every 8 connection(s)
I20260926 20:43:50.839892 2882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
Sep 26 20:43:50 dist-test-slave-4hd3 krb5kdc[2775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:50.847366 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 2882
I20260926 20:43:50.847471 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:43:50.847651 393 external_mini_cluster.cc:1428] Setting key 769621ea8a393bd3df771b49ba11f7d4
I20260926 20:43:50.849180 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:50.840716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:34587 (local address 127.0.98.126:33721)
0926 20:43:50.840804 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:50.840806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:50.840977 (+ 171us) server_negotiation.cc:407] Connection header received
0926 20:43:50.841874 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:50.841875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:50.841890 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:50.841942 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:50.842528 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:50.843021 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:50.844097 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:50.844310 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:50.844374 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:50.846364 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:50.846365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:50.846366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:50.846372 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:50.847650 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:50.848159 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:50.848160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:50.848160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:50.848220 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:50.848493 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:50.848494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:50.848494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:50.848647 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:50.848724 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:50.848995 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 20:43:50.849080 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:43:50.849377 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:50.840628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33721 (local address 127.0.98.65:34587)
0926 20:43:50.840863 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:50.840897 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:50.841763 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:50.842002 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:50.842005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:50.842271 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:50.842463 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:50.842467 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:50.843096 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:50.843097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:50.844016 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:50.844018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:50.844388 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:50.844389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:50.844483 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:50.844919 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:50.844926 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:50.846277 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:50.847740 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:50.847747 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:50.847749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:50.848065 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:50.848285 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:50.848285 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:50.848286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:50.848429 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:50.848727 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:50.848733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:50.848867 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 20:43:50.848974 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 20:43:50.849141 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":87,"threads_started":1}
I20260926 20:43:50.850031 3011 heartbeater.cc:347] Connected to a master server at 127.0.98.126:33721
I20260926 20:43:50.850109 3011 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:50.850315 3011 heartbeater.cc:511] Master 127.0.98.126:33721 requested a full tablet report, sending...
I20260926 20:43:50.850838 2820 ts_manager.cc:194] Registered new tserver with Master: dda976e1faf04fb1b8d52aafe7d57cfb (127.0.98.65:40491)
I20260926 20:43:50.851420 2820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:34587
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260926 20:43:50.872184 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33721
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:50.953969 3016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:50.954180 3016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:50.954203 3016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:50.955365 3016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:50.955410 3016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:50.955435 3016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:43:50.957053 3016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:50.957343 3016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:50.957579 3016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:50.962499 3022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:50.962487 3024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:50.962502 3021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:50.962911 3016 server_base.cc:1034] running on GCE node
I20260926 20:43:50.963143 3016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:50.963399 3016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:50.964581 3016 hybrid_clock.cc:648] HybridClock initialized: now 1790455430964565 us; error 29 us; skew 500 ppm
Sep 26 20:43:50 dist-test-slave-4hd3 krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:50.967098 3016 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:43:50.967597 3016 webserver.cc:460] Webserver started at http://127.0.98.66:35261/ using document root <none> and password file <none>
I20260926 20:43:50.967876 3016 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:50.967947 3016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:50.968108 3016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:50.969197 3016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "d024a84288274ae0b3b007e006e2f42e"
format_stamp: "Formatted at 2026-09-26 20:43:50 on dist-test-slave-4hd3"
server_key: "bfe844d1792f44891c9396ff1cc9e9d9"
server_key_iv: "8b05f749db2248ceba46c8627bcdabab"
server_key_version: "encryptionkey@0"
I20260926 20:43:50.969588 3016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "d024a84288274ae0b3b007e006e2f42e"
format_stamp: "Formatted at 2026-09-26 20:43:50 on dist-test-slave-4hd3"
server_key: "bfe844d1792f44891c9396ff1cc9e9d9"
server_key_iv: "8b05f749db2248ceba46c8627bcdabab"
server_key_version: "encryptionkey@0"
I20260926 20:43:50.970976 3016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:43:50.971829 3031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:50.972086 3016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:43:50.972221 3016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "d024a84288274ae0b3b007e006e2f42e"
format_stamp: "Formatted at 2026-09-26 20:43:50 on dist-test-slave-4hd3"
server_key: "bfe844d1792f44891c9396ff1cc9e9d9"
server_key_iv: "8b05f749db2248ceba46c8627bcdabab"
server_key_version: "encryptionkey@0"
I20260926 20:43:50.972312 3016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:50.997198 3016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:50.999627 3016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:50.999822 3016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:51.000069 3016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:51.000418 3016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:51.000478 3016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:51.000519 3016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:51.000547 3016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:51.006397 3016 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:42665
I20260926 20:43:51.006455 3144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:42665 every 8 connection(s)
I20260926 20:43:51.006783 3016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:43:51 dist-test-slave-4hd3 krb5kdc[2775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:51.015482 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:51.007496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:43899 (local address 127.0.98.126:33721)
0926 20:43:51.007579 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:51.007582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:51.007736 (+ 154us) server_negotiation.cc:407] Connection header received
0926 20:43:51.008688 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:51.008689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:51.008700 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:51.008779 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:51.009338 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:51.009836 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:51.010816 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:51.010992 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:51.011043 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:51.013073 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:51.013074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:51.013075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:51.013081 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:51.014337 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:51.014696 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:51.014697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:51.014698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:51.014746 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:51.014950 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:51.014950 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:51.014951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:51.015045 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:51.015093 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:51.015349 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 20:43:51.015420 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:43:51.015550 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:51.007447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33721 (local address 127.0.98.66:43899)
0926 20:43:51.007600 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:51.007637 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:51.008599 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:51.008833 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:51.008837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:51.009072 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:51.009265 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:51.009269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:51.009924 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:51.009925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:51.010744 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:51.010746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:51.011060 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:51.011060 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:51.011166 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:51.011670 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:51.011678 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:51.012990 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:51.014410 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:51.014415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:51.014417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:51.014628 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:51.014805 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:51.014806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:51.014806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:51.014899 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:51.015106 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:51.015110 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:51.015213 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 20:43:51.015292 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:43:51.015400 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":95,"threads_started":1}
I20260926 20:43:51.016111 3145 heartbeater.cc:347] Connected to a master server at 127.0.98.126:33721
I20260926 20:43:51.016193 3145 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:51.016382 3145 heartbeater.cc:511] Master 127.0.98.126:33721 requested a full tablet report, sending...
I20260926 20:43:51.016769 2820 ts_manager.cc:194] Registered new tserver with Master: d024a84288274ae0b3b007e006e2f42e (127.0.98.66:42665)
I20260926 20:43:51.016832 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 3016
I20260926 20:43:51.016937 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:43:51.017138 393 external_mini_cluster.cc:1428] Setting key 95c26efb53056ea336b9bcd536e3c3f3
I20260926 20:43:51.017227 2820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:43899
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260926 20:43:51.042071 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33721
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:51.122725 3150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:51.122965 3150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:51.122988 3150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:51.124152 3150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:51.124193 3150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:51.124219 3150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:43:51.125622 3150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:51.125808 3150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:51.125970 3150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:51.130868 3156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:51.130863 3155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:51.130863 3158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:51.131094 3150 server_base.cc:1034] running on GCE node
I20260926 20:43:51.131340 3150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:51.131582 3150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:51.132761 3150 hybrid_clock.cc:648] HybridClock initialized: now 1790455431132748 us; error 52 us; skew 500 ppm
Sep 26 20:43:51 dist-test-slave-4hd3 krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:51.135191 3150 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:43:51.135625 3150 webserver.cc:460] Webserver started at http://127.0.98.67:40847/ using document root <none> and password file <none>
I20260926 20:43:51.135916 3150 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:51.135987 3150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:51.136133 3150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:51.137225 3150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "4c7e652cc633472088c35da14996d4f8"
format_stamp: "Formatted at 2026-09-26 20:43:51 on dist-test-slave-4hd3"
server_key: "b385b24db8c549cde35d73e9ffc5da3d"
server_key_iv: "6ba7aa8b3d03356b5e9b3c47481a2833"
server_key_version: "encryptionkey@0"
I20260926 20:43:51.137578 3150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "4c7e652cc633472088c35da14996d4f8"
format_stamp: "Formatted at 2026-09-26 20:43:51 on dist-test-slave-4hd3"
server_key: "b385b24db8c549cde35d73e9ffc5da3d"
server_key_iv: "6ba7aa8b3d03356b5e9b3c47481a2833"
server_key_version: "encryptionkey@0"
I20260926 20:43:51.138955 3150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:43:51.139859 3165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:51.140043 3150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:43:51.140147 3150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "4c7e652cc633472088c35da14996d4f8"
format_stamp: "Formatted at 2026-09-26 20:43:51 on dist-test-slave-4hd3"
server_key: "b385b24db8c549cde35d73e9ffc5da3d"
server_key_iv: "6ba7aa8b3d03356b5e9b3c47481a2833"
server_key_version: "encryptionkey@0"
I20260926 20:43:51.140241 3150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:51.147970 3150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:51.149921 3150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:51.150085 3150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:51.150305 3150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:51.150609 3150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:51.150669 3150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:51.150710 3150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:51.150765 3150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:51.156512 3150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:45381
I20260926 20:43:51.156610 3278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:45381 every 8 connection(s)
I20260926 20:43:51.156910 3150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:43:51 dist-test-slave-4hd3 krb5kdc[2775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:51.166361 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:51.157644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:54819 (local address 127.0.98.126:33721)
0926 20:43:51.157691 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:51.157693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:51.157892 (+ 199us) server_negotiation.cc:407] Connection header received
0926 20:43:51.158936 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:51.158938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:51.158951 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:51.159002 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:51.159663 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:51.160198 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:51.161205 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:51.161404 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:51.161491 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:51.163599 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:51.163601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:51.163601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:51.163616 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:51.164941 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:51.165387 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:51.165389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:51.165389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:51.165448 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:51.165721 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:51.165722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:51.165722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:51.165835 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:51.165915 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:51.166179 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 20:43:51.166262 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:43:51.166550 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:51.157624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33721 (local address 127.0.98.67:54819)
0926 20:43:51.157769 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:51.157804 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:51.158800 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:51.159090 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:51.159096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:51.159363 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:51.159555 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:51.159559 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:51.160295 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:51.160296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:51.161098 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:51.161100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:51.161502 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:51.161503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:51.161635 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:51.162104 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:51.162113 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:51.163497 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:51.165052 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:51.165059 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:51.165061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:51.165298 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:51.165515 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:51.165515 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:51.165516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:51.165622 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:51.165906 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:51.165913 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:51.166049 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 20:43:51.166158 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 20:43:51.166322 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260926 20:43:51.166826 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 3150
I20260926 20:43:51.166923 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:43:51.167135 393 external_mini_cluster.cc:1428] Setting key 99af986792ef63e7c97759c3d5eff017
I20260926 20:43:51.167277 3279 heartbeater.cc:347] Connected to a master server at 127.0.98.126:33721
I20260926 20:43:51.167388 3279 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:51.167579 3279 heartbeater.cc:511] Master 127.0.98.126:33721 requested a full tablet report, sending...
I20260926 20:43:51.168071 2820 ts_manager.cc:194] Registered new tserver with Master: 4c7e652cc633472088c35da14996d4f8 (127.0.98.67:45381)
I20260926 20:43:51.168468 2820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:54819
I20260926 20:43:51.179009 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 20:43:51.189777 2860 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:43:51.188706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59732 (local address 127.0.98.126:33721)
0926 20:43:51.188804 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:51.188807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:51.188917 (+ 110us) server_negotiation.cc:407] Connection header received
0926 20:43:51.189011 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:51.189012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:51.189022 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:51.189086 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:51.189695 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59732: BlockingRecv error: recv got EOF from 127.0.0.1:59732 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:43:51.190543 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 2882
I20260926 20:43:51.195182 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 3016
I20260926 20:43:51.199419 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 3150
I20260926 20:43:51.204042 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 2788
2026-09-26T20:43:51Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1903 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:43:51 dist-test-slave-4hd3 krb5kdc[3291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:43:51 dist-test-slave-4hd3 krb5kdc[3291](info): set up 2 sockets
Sep 26 20:43:51 dist-test-slave-4hd3 krb5kdc[3291](info): commencing operation
krb5kdc: starting...
W20260926 20:43:52.236107 393 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:43:52 dist-test-slave-4hd3 krb5kdc[3291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:43:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:43:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:43:52.348733 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33011
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:33011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:52.426960 3304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:52.427187 3304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:52.427227 3304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:52.428435 3304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:43:52.428501 3304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:52.428526 3304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:52.428555 3304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:43:52.428586 3304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:43:52.430070 3304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:33011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:52.430351 3304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:52.430573 3304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:52.435322 3309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:52.435308 3310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:52.435312 3312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:52.435513 3304 server_base.cc:1034] running on GCE node
I20260926 20:43:52.435873 3304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:52.436107 3304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:52.437273 3304 hybrid_clock.cc:648] HybridClock initialized: now 1790455432437265 us; error 39 us; skew 500 ppm
Sep 26 20:43:52 dist-test-slave-4hd3 krb5kdc[3291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:52.439780 3304 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:43:52.440186 3304 webserver.cc:460] Webserver started at http://127.0.98.126:45665/ using document root <none> and password file <none>
I20260926 20:43:52.440415 3304 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:52.440483 3304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:52.440632 3304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:52.441655 3304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "2dfd6d04691e421991dfb4e2791b4c74"
format_stamp: "Formatted at 2026-09-26 20:43:52 on dist-test-slave-4hd3"
server_key: "9b4f8f19abb26b5ddb6d25d003f684d5"
server_key_iv: "fca5fca2437cd4894fd45047b9a6ad6f"
server_key_version: "encryptionkey@0"
I20260926 20:43:52.442013 3304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "2dfd6d04691e421991dfb4e2791b4c74"
format_stamp: "Formatted at 2026-09-26 20:43:52 on dist-test-slave-4hd3"
server_key: "9b4f8f19abb26b5ddb6d25d003f684d5"
server_key_iv: "fca5fca2437cd4894fd45047b9a6ad6f"
server_key_version: "encryptionkey@0"
I20260926 20:43:52.443411 3304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:43:52.444264 3319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:52.444442 3304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:43:52.444545 3304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "2dfd6d04691e421991dfb4e2791b4c74"
format_stamp: "Formatted at 2026-09-26 20:43:52 on dist-test-slave-4hd3"
server_key: "9b4f8f19abb26b5ddb6d25d003f684d5"
server_key_iv: "fca5fca2437cd4894fd45047b9a6ad6f"
server_key_version: "encryptionkey@0"
I20260926 20:43:52.444617 3304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:52.458338 3304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:52.460337 3304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:52.460536 3304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:52.464483 3304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:33011
I20260926 20:43:52.464521 3371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:33011 every 8 connection(s)
I20260926 20:43:52.464891 3304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:43:52.465718 3372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:52.468007 3372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74: Bootstrap starting.
I20260926 20:43:52.468605 3372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:52.468914 3372 log.cc:824] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:52.469578 3372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74: No bootstrap required, opened a new log
I20260926 20:43:52.471146 3372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfd6d04691e421991dfb4e2791b4c74" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33011 } }
I20260926 20:43:52.471303 3372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:52.471328 3372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dfd6d04691e421991dfb4e2791b4c74, State: Initialized, Role: FOLLOWER
I20260926 20:43:52.471509 3372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfd6d04691e421991dfb4e2791b4c74" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33011 } }
I20260926 20:43:52.471572 3372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:43:52.471616 3372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:43:52.471665 3372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:52.472596 3372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfd6d04691e421991dfb4e2791b4c74" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33011 } }
I20260926 20:43:52.472716 3372 leader_election.cc:302] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dfd6d04691e421991dfb4e2791b4c74; no voters:
I20260926 20:43:52.472905 3372 leader_election.cc:288] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:43:52.472957 3375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:52.473202 3372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:43:52.473292 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 3304
I20260926 20:43:52.473380 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:43:52.473551 393 external_mini_cluster.cc:1428] Setting key b165a53381984177f1470ffa29dcaeff
I20260926 20:43:52.473672 3375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74 [term 1 LEADER]: Becoming Leader. State: Replica: 2dfd6d04691e421991dfb4e2791b4c74, State: Running, Role: LEADER
I20260926 20:43:52.473809 3375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfd6d04691e421991dfb4e2791b4c74" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33011 } }
I20260926 20:43:52.474442 3377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dfd6d04691e421991dfb4e2791b4c74. Latest consensus state: current_term: 1 leader_uuid: "2dfd6d04691e421991dfb4e2791b4c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfd6d04691e421991dfb4e2791b4c74" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33011 } } }
I20260926 20:43:52.474531 3377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74 [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:52.474884 3376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dfd6d04691e421991dfb4e2791b4c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfd6d04691e421991dfb4e2791b4c74" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33011 } } }
I20260926 20:43:52.474961 3376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74 [sys.catalog]: This master's current role is: LEADER
W20260926 20:43:52.475185 3391 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:43:52.475291 3391 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:43:52.475394 3392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:43:52.475589 3392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:43:52.477236 3392 catalog_manager.cc:1347] Generated new cluster ID: cf2a7923401a44eabe3f6bb56b8d08cd
I20260926 20:43:52.477295 3392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:43:52 dist-test-slave-4hd3 krb5kdc[3291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:52.486485 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:52.474039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42740 (local address 127.0.98.126:33011)
0926 20:43:52.474964 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:52.474974 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:52.475010 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:43:52.475140 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:52.475147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:52.475221 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:52.476203 (+ 982us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:52.476297 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:52.477666 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:52.478841 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:52.479513 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:52.479630 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:52.482562 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:52.482567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:52.482567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:52.482581 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:52.484681 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:52.485092 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:52.485095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:52.485096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:52.485206 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:52.485415 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:52.485416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:52.485417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:52.485733 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:52.485906 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:52.486003 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 20:43:52.486251 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":85,"threads_started":1}
I20260926 20:43:52.487484 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:43:52.488911 3392 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:43:52.489709 3392 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:43:52.496333 3392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dfd6d04691e421991dfb4e2791b4c74: Generated new TSK 0
I20260926 20:43:52.496512 3392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:43:52.510790 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:52.506716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.0.98.126:33011)
0926 20:43:52.506777 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:52.506779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:52.506903 (+ 124us) server_negotiation.cc:407] Connection header received
0926 20:43:52.506951 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:52.506952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:52.506964 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:52.507038 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:52.507433 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:52.508292 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:52.509075 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:52.509414 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:52.509486 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:52.510219 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:52.510220 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:52.510221 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:43:52.510446 (+ 225us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:52.510464 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:43:52.510471 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:52.510532 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:52.510683 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 20:43:52.510708 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:43:52.513461 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 3304
2026-09-26T20:43:52Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1307 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:43:52 dist-test-slave-4hd3 krb5kdc[3405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:43:52 dist-test-slave-4hd3 krb5kdc[3405](info): set up 2 sockets
Sep 26 20:43:52 dist-test-slave-4hd3 krb5kdc[3405](info): commencing operation
krb5kdc: starting...
W20260926 20:43:53.543606 393 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:43:53 dist-test-slave-4hd3 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:43:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:43:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:43:53.658399 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:46285
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:46285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:53.738121 3418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:53.738335 3418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:53.738358 3418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:53.739539 3418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:43:53.739581 3418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:53.739596 3418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:53.739609 3418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:43:53.739662 3418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:43:53.741163 3418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:46285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:46285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:53.741413 3418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:53.741644 3418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:53.746661 3426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:53.746621 3424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:53.746716 3418 server_base.cc:1034] running on GCE node
W20260926 20:43:53.746635 3423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:53.747247 3418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:53.747573 3418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:53.748737 3418 hybrid_clock.cc:648] HybridClock initialized: now 1790455433748721 us; error 52 us; skew 500 ppm
Sep 26 20:43:53 dist-test-slave-4hd3 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:53.751195 3418 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:43:53.751704 3418 webserver.cc:460] Webserver started at http://127.0.98.126:44491/ using document root <none> and password file <none>
I20260926 20:43:53.751991 3418 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:53.752061 3418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:53.752213 3418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:53.753322 3418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "a8df0373f2e04fabacb3751bb2552064"
format_stamp: "Formatted at 2026-09-26 20:43:53 on dist-test-slave-4hd3"
server_key: "740913960865897647f1165f21a32e74"
server_key_iv: "333735e774081006a5630b684362e352"
server_key_version: "encryptionkey@0"
I20260926 20:43:53.753691 3418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "a8df0373f2e04fabacb3751bb2552064"
format_stamp: "Formatted at 2026-09-26 20:43:53 on dist-test-slave-4hd3"
server_key: "740913960865897647f1165f21a32e74"
server_key_iv: "333735e774081006a5630b684362e352"
server_key_version: "encryptionkey@0"
I20260926 20:43:53.755010 3418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:43:53.755805 3433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:53.755991 3418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:43:53.756089 3418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "a8df0373f2e04fabacb3751bb2552064"
format_stamp: "Formatted at 2026-09-26 20:43:53 on dist-test-slave-4hd3"
server_key: "740913960865897647f1165f21a32e74"
server_key_iv: "333735e774081006a5630b684362e352"
server_key_version: "encryptionkey@0"
I20260926 20:43:53.756206 3418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:53.772163 3418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:53.774015 3418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:53.774190 3418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:53.778869 3418 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:46285
I20260926 20:43:53.778908 3485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:46285 every 8 connection(s)
I20260926 20:43:53.779261 3418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:43:53.780333 3486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:53.782550 3486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064: Bootstrap starting.
I20260926 20:43:53.782847 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 3418
I20260926 20:43:53.782950 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:43:53.783107 393 external_mini_cluster.cc:1428] Setting key 5e2339bc224fa35c6ddb3c750b89045e
I20260926 20:43:53.783123 3486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:53.783411 3486 log.cc:824] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:53.784154 3486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064: No bootstrap required, opened a new log
I20260926 20:43:53.785934 3486 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8df0373f2e04fabacb3751bb2552064" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46285 } }
I20260926 20:43:53.786069 3486 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:53.786091 3486 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8df0373f2e04fabacb3751bb2552064, State: Initialized, Role: FOLLOWER
I20260926 20:43:53.786237 3486 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8df0373f2e04fabacb3751bb2552064" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46285 } }
I20260926 20:43:53.786335 3486 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:43:53.786366 3486 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:43:53.786417 3486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:53.787102 3486 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8df0373f2e04fabacb3751bb2552064" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46285 } }
I20260926 20:43:53.787228 3486 leader_election.cc:302] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8df0373f2e04fabacb3751bb2552064; no voters:
I20260926 20:43:53.787426 3486 leader_election.cc:288] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:43:53.787456 3491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:53.787585 3491 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064 [term 1 LEADER]: Becoming Leader. State: Replica: a8df0373f2e04fabacb3751bb2552064, State: Running, Role: LEADER
I20260926 20:43:53.787705 3486 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:43:53.787815 3491 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8df0373f2e04fabacb3751bb2552064" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46285 } }
I20260926 20:43:53.788335 3492 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8df0373f2e04fabacb3751bb2552064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8df0373f2e04fabacb3751bb2552064" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46285 } } }
I20260926 20:43:53.788431 3492 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064 [sys.catalog]: This master's current role is: LEADER
Sep 26 20:43:53 dist-test-slave-4hd3 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:53.789424 3493 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8df0373f2e04fabacb3751bb2552064. Latest consensus state: current_term: 1 leader_uuid: "a8df0373f2e04fabacb3751bb2552064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8df0373f2e04fabacb3751bb2552064" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46285 } } }
W20260926 20:43:53.789549 3507 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 20:43:53.790617 3493 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064 [sys.catalog]: This master's current role is: LEADER
W20260926 20:43:53.790658 3507 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:43:53.790745 3500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:43:53.790966 3500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:43:53.792487 3500 catalog_manager.cc:1347] Generated new cluster ID: bb48375b4e9249658b1b510fea29d548
I20260926 20:43:53.792541 3500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:43:53.795317 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:53.783604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42474 (local address 127.0.98.126:46285)
0926 20:43:53.783807 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:53.783817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:53.783854 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:43:53.784329 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:53.784336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:53.784385 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:53.784534 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:53.785160 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:53.786044 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:53.787229 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:53.787920 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:53.788028 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:53.791212 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:53.791215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:53.791215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:53.791227 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:53.793357 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:53.793762 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:53.793765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:53.793766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:53.793866 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:53.794064 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:53.794065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:53.794065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:53.794368 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:53.794522 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:53.794962 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 20:43:53.795096 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
I20260926 20:43:53.796367 3500 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:43:53.796836 3500 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:43:53.802063 3500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8df0373f2e04fabacb3751bb2552064: Generated new TSK 0
I20260926 20:43:53.802259 3500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260926 20:43:53.834012 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46285
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:53.935791 3512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:53.935988 3512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:53.936012 3512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:53.937184 3512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:53.937229 3512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:53.937255 3512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:43:53.938737 3512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:53.938962 3512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:53.939208 3512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:53.944008 3520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:53.944046 3517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:53.944046 3518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:53.944274 3512 server_base.cc:1034] running on GCE node
I20260926 20:43:53.944468 3512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:53.944711 3512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:53.945892 3512 hybrid_clock.cc:648] HybridClock initialized: now 1790455433945870 us; error 44 us; skew 500 ppm
Sep 26 20:43:53 dist-test-slave-4hd3 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:53.948469 3512 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:43:53.949007 3512 webserver.cc:460] Webserver started at http://127.0.98.65:33965/ using document root <none> and password file <none>
I20260926 20:43:53.949229 3512 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:53.949295 3512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:53.949437 3512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:53.950505 3512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "600b930f1dc64ec98c87f67a97af6b74"
format_stamp: "Formatted at 2026-09-26 20:43:53 on dist-test-slave-4hd3"
server_key: "8c44b819bf507545d16dbd53075e2005"
server_key_iv: "8a9475a714fdd7db6d50b71700e278fb"
server_key_version: "encryptionkey@0"
I20260926 20:43:53.950881 3512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "600b930f1dc64ec98c87f67a97af6b74"
format_stamp: "Formatted at 2026-09-26 20:43:53 on dist-test-slave-4hd3"
server_key: "8c44b819bf507545d16dbd53075e2005"
server_key_iv: "8a9475a714fdd7db6d50b71700e278fb"
server_key_version: "encryptionkey@0"
I20260926 20:43:53.952338 3512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 20:43:53.953182 3527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:53.953392 3512 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:43:53.953501 3512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "600b930f1dc64ec98c87f67a97af6b74"
format_stamp: "Formatted at 2026-09-26 20:43:53 on dist-test-slave-4hd3"
server_key: "8c44b819bf507545d16dbd53075e2005"
server_key_iv: "8a9475a714fdd7db6d50b71700e278fb"
server_key_version: "encryptionkey@0"
I20260926 20:43:53.953576 3512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:53.976001 3512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:53.978415 3512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:53.978579 3512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:53.978838 3512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:53.979151 3512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:53.979208 3512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:53.979249 3512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:53.979279 3512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:53.985055 3512 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:43739
I20260926 20:43:53.985095 3640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:43739 every 8 connection(s)
I20260926 20:43:53.985427 3512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:43:53.989542 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 3512
I20260926 20:43:53.989619 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:43:53.989842 393 external_mini_cluster.cc:1428] Setting key a66e9233957a5f6ffb4797792d740a2f
Sep 26 20:43:53 dist-test-slave-4hd3 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:53.995394 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:53.986118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:34471 (local address 127.0.98.126:46285)
0926 20:43:53.986216 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:53.986218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:53.986438 (+ 220us) server_negotiation.cc:407] Connection header received
0926 20:43:53.986954 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:53.986955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:53.986966 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:53.987020 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:53.988027 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:53.988610 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:53.989801 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:53.990107 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:53.990169 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:53.992533 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:53.992535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:53.992535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:53.992543 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:53.994020 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:53.994433 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:53.994435 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:53.994435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:53.994493 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:53.994748 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:53.994749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:53.994749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:53.994867 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:53.994946 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:53.995222 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 20:43:53.995306 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:43:53.995587 3643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:53.986104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46285 (local address 127.0.98.65:34471)
0926 20:43:53.986314 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:53.986351 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:53.986870 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:53.987140 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:53.987144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:53.987736 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:53.987925 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:53.987930 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:53.988730 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:53.988731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:53.989707 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:53.989709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:53.990174 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:53.990175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:53.990310 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:53.990848 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:53.990855 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:53.992437 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:53.994126 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:53.994132 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:53.994135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:53.994358 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:53.994568 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:53.994569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:53.994569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:53.994677 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:53.994952 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:53.994958 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:53.995090 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 20:43:53.995197 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 20:43:53.995350 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":130,"threads_started":1}
I20260926 20:43:53.996114 3641 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46285
I20260926 20:43:53.996234 3641 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:53.996407 3641 heartbeater.cc:511] Master 127.0.98.126:46285 requested a full tablet report, sending...
I20260926 20:43:53.997004 3450 ts_manager.cc:194] Registered new tserver with Master: 600b930f1dc64ec98c87f67a97af6b74 (127.0.98.65:43739)
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260926 20:43:53.997936 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:34471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260926 20:43:54.014626 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46285
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:54.095149 3646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:54.095350 3646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:54.095373 3646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:54.096509 3646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:54.096547 3646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:54.096572 3646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:43:54.097889 3646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:54.098066 3646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:54.098227 3646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:54.103166 3651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:54.103185 3654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:54.103241 3652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:54.103277 3646 server_base.cc:1034] running on GCE node
I20260926 20:43:54.103582 3646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:54.103893 3646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:54.105062 3646 hybrid_clock.cc:648] HybridClock initialized: now 1790455434105060 us; error 38 us; skew 500 ppm
Sep 26 20:43:54 dist-test-slave-4hd3 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:54.107515 3646 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:43:54.108002 3646 webserver.cc:460] Webserver started at http://127.0.98.66:46001/ using document root <none> and password file <none>
I20260926 20:43:54.108223 3646 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:54.108299 3646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:54.108454 3646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:54.109519 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "bb1255316f1f4682a83fa65f662218ef"
format_stamp: "Formatted at 2026-09-26 20:43:54 on dist-test-slave-4hd3"
server_key: "6108848b1d58bb74359e0d2058173a16"
server_key_iv: "d2eda6763ce566f4e907b509753f5a68"
server_key_version: "encryptionkey@0"
I20260926 20:43:54.109875 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "bb1255316f1f4682a83fa65f662218ef"
format_stamp: "Formatted at 2026-09-26 20:43:54 on dist-test-slave-4hd3"
server_key: "6108848b1d58bb74359e0d2058173a16"
server_key_iv: "d2eda6763ce566f4e907b509753f5a68"
server_key_version: "encryptionkey@0"
I20260926 20:43:54.111181 3646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:43:54.111933 3661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:54.112103 3646 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:43:54.112198 3646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "bb1255316f1f4682a83fa65f662218ef"
format_stamp: "Formatted at 2026-09-26 20:43:54 on dist-test-slave-4hd3"
server_key: "6108848b1d58bb74359e0d2058173a16"
server_key_iv: "d2eda6763ce566f4e907b509753f5a68"
server_key_version: "encryptionkey@0"
I20260926 20:43:54.112293 3646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:54.123024 3646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:54.124902 3646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:54.125061 3646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:54.125280 3646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:54.125574 3646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:54.125627 3646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:54.125666 3646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:54.125694 3646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:54.131421 3646 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:43055
I20260926 20:43:54.131472 3774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:43055 every 8 connection(s)
I20260926 20:43:54.131844 3646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:43:54 dist-test-slave-4hd3 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:54.139083 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 3646
I20260926 20:43:54.139160 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:43:54.139338 393 external_mini_cluster.cc:1428] Setting key 4b22aea13772915e1fb4270a723d103c
I20260926 20:43:54.140880 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:54.132532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:57361 (local address 127.0.98.126:46285)
0926 20:43:54.132580 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:54.132583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:54.132798 (+ 215us) server_negotiation.cc:407] Connection header received
0926 20:43:54.133256 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:54.133257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:54.133267 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:54.133318 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:54.134170 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:54.134644 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:54.135667 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:54.135953 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:54.136012 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:54.137954 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:54.137955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:54.137956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:54.137963 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:54.139184 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:54.139664 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:54.139665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:54.139666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:54.139768 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:54.140160 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:54.140161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:54.140162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:54.140305 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:54.140370 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:54.140730 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 20:43:54.140817 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:43:54.141168 3777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:54.132529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46285 (local address 127.0.98.66:57361)
0926 20:43:54.132679 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:54.132712 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:54.133176 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:54.133369 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:54.133372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:54.133921 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:54.134100 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:54.134105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:54.134743 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:54.134743 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:54.135574 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:54.135576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:54.136029 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:54.136030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:54.136125 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:54.136557 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:54.136564 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:54.137870 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:54.139290 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:54.139297 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:54.139299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:54.139574 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:54.139842 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:54.139843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:54.139843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:54.139957 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:54.140383 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:54.140388 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:54.140575 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 20:43:54.140690 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 20:43:54.140902 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":91,"threads_started":1}
I20260926 20:43:54.141683 3775 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46285
I20260926 20:43:54.141825 3775 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:54.142012 3775 heartbeater.cc:511] Master 127.0.98.126:46285 requested a full tablet report, sending...
I20260926 20:43:54.142424 3450 ts_manager.cc:194] Registered new tserver with Master: bb1255316f1f4682a83fa65f662218ef (127.0.98.66:43055)
I20260926 20:43:54.142803 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:57361
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260926 20:43:54.163122 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46285
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:54.243314 3780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:54.243554 3780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:54.243592 3780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:54.244791 3780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:54.244863 3780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:54.244907 3780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:43:54.246385 3780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:54.246654 3780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:54.246888 3780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:43:54.251922 3780 server_base.cc:1034] running on GCE node
W20260926 20:43:54.251935 3788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:54.251938 3786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:54.252039 3785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:54.252327 3780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:54.252573 3780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:54.253738 3780 hybrid_clock.cc:648] HybridClock initialized: now 1790455434253719 us; error 28 us; skew 500 ppm
Sep 26 20:43:54 dist-test-slave-4hd3 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:54.256254 3780 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:43:54.256635 3780 webserver.cc:460] Webserver started at http://127.0.98.67:40151/ using document root <none> and password file <none>
I20260926 20:43:54.256853 3780 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:54.256896 3780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:54.256983 3780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:54.258034 3780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "e1adee9fbffb472d9cf432216e713972"
format_stamp: "Formatted at 2026-09-26 20:43:54 on dist-test-slave-4hd3"
server_key: "5e648e0623ecb24bb04ddd9514ac2727"
server_key_iv: "e4a217cd9fcf0ef8030fb5bb8d13be02"
server_key_version: "encryptionkey@0"
I20260926 20:43:54.258404 3780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "e1adee9fbffb472d9cf432216e713972"
format_stamp: "Formatted at 2026-09-26 20:43:54 on dist-test-slave-4hd3"
server_key: "5e648e0623ecb24bb04ddd9514ac2727"
server_key_iv: "e4a217cd9fcf0ef8030fb5bb8d13be02"
server_key_version: "encryptionkey@0"
I20260926 20:43:54.259663 3780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:43:54.260376 3795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:54.260542 3780 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:43:54.260628 3780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "e1adee9fbffb472d9cf432216e713972"
format_stamp: "Formatted at 2026-09-26 20:43:54 on dist-test-slave-4hd3"
server_key: "5e648e0623ecb24bb04ddd9514ac2727"
server_key_iv: "e4a217cd9fcf0ef8030fb5bb8d13be02"
server_key_version: "encryptionkey@0"
I20260926 20:43:54.260699 3780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:54.271643 3780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:54.273778 3780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:54.273936 3780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:54.274173 3780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:54.274492 3780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:54.274533 3780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:54.274574 3780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:54.274601 3780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:54.280731 3780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:44677
I20260926 20:43:54.280789 3908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:44677 every 8 connection(s)
I20260926 20:43:54.281090 3780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:43:54 dist-test-slave-4hd3 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:54.287747 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 3780
I20260926 20:43:54.287840 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:43:54.288087 393 external_mini_cluster.cc:1428] Setting key 744ea42c09c698619a67f7bf3e860d0d
I20260926 20:43:54.290710 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:54.281786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:53119 (local address 127.0.98.126:46285)
0926 20:43:54.281832 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:54.281834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:54.282027 (+ 193us) server_negotiation.cc:407] Connection header received
0926 20:43:54.282577 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:54.282578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:54.282589 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:54.282640 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:54.283569 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:54.284219 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:54.285220 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:54.285416 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:54.285495 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:54.287554 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:54.287556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:54.287556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:54.287564 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:54.289562 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:54.289969 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:54.289970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:54.289970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:54.290021 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:54.290241 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:54.290242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:54.290242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:54.290339 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:54.290389 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:54.290595 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 20:43:54.290648 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:43:54.290999 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:54.281762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46285 (local address 127.0.98.67:53119)
0926 20:43:54.281907 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:54.281947 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:54.282456 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:54.282717 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:54.282721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:54.283282 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:54.283467 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:54.283471 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:54.284321 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:54.284321 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:54.285122 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:54.285124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:54.285502 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:54.285503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:54.285632 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:54.286076 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:54.286085 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:54.287434 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:54.289639 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:54.289685 (+ 46us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:54.289687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:54.289901 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:54.290079 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:54.290080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:54.290080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:54.290181 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:54.290393 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:54.290396 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:54.290505 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 20:43:54.290585 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 20:43:54.290741 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260926 20:43:54.291461 3909 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46285
I20260926 20:43:54.291555 3909 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:54.291684 3909 heartbeater.cc:511] Master 127.0.98.126:46285 requested a full tablet report, sending...
I20260926 20:43:54.292098 3450 ts_manager.cc:194] Registered new tserver with Master: e1adee9fbffb472d9cf432216e713972 (127.0.98.67:44677)
I20260926 20:43:54.292420 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:53119
I20260926 20:43:54.300105 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:43:54 dist-test-slave-4hd3 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455434, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 20:43:54 dist-test-slave-4hd3 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455434, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:54.322556 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:54.315501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42484 (local address 127.0.98.126:46285)
0926 20:43:54.315564 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:54.315567 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:54.315743 (+ 176us) server_negotiation.cc:407] Connection header received
0926 20:43:54.315855 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:54.315856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:54.315873 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:54.315939 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:54.316495 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:54.316988 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:54.317717 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:54.317921 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:54.318005 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:54.320191 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:54.320193 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:54.320193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:54.320201 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:54.321425 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:54.321755 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:54.321756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:54.321757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:54.321816 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:54.322020 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:54.322021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:54.322021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:54.322169 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:54.322274 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:54.322373 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 20:43:54.322452 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 20:43:54.322829 3450 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42484
I20260926 20:43:54.323279 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 3512
I20260926 20:43:54.327411 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 3646
I20260926 20:43:54.331561 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 3780
I20260926 20:43:54.335779 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 3418
2026-09-26T20:43:54Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1823 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:43:54 dist-test-slave-4hd3 krb5kdc[3921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:43:54 dist-test-slave-4hd3 krb5kdc[3921](info): set up 2 sockets
Sep 26 20:43:54 dist-test-slave-4hd3 krb5kdc[3921](info): commencing operation
krb5kdc: starting...
W20260926 20:43:55.383574 393 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:43:55 dist-test-slave-4hd3 krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:43:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:43:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:43:55.499589 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:35799
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:35799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:55.579348 3934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:55.579571 3934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:55.579594 3934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:55.580907 3934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:43:55.580956 3934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:55.580971 3934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:55.580984 3934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:43:55.580996 3934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:43:55.582399 3934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:35799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:35799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:55.582584 3934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:55.582767 3934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:55.588537 3940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:55.588603 3934 server_base.cc:1034] running on GCE node
W20260926 20:43:55.588546 3939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:55.588546 3942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:55.589076 3934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:55.589538 3934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:55.590680 3934 hybrid_clock.cc:648] HybridClock initialized: now 1790455435590655 us; error 35 us; skew 500 ppm
Sep 26 20:43:55 dist-test-slave-4hd3 krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:55.594480 3934 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:43:55.595068 3934 webserver.cc:460] Webserver started at http://127.0.98.126:38443/ using document root <none> and password file <none>
I20260926 20:43:55.595361 3934 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:55.595431 3934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:55.595573 3934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:55.597111 3934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "29b11a4ac3b5434ca6d1ed95d0c38456"
format_stamp: "Formatted at 2026-09-26 20:43:55 on dist-test-slave-4hd3"
server_key: "75a1fc636e6d7b92f07d3b3bba24a97e"
server_key_iv: "819a579e3cbfb15aed94916dd015178a"
server_key_version: "encryptionkey@0"
I20260926 20:43:55.597484 3934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "29b11a4ac3b5434ca6d1ed95d0c38456"
format_stamp: "Formatted at 2026-09-26 20:43:55 on dist-test-slave-4hd3"
server_key: "75a1fc636e6d7b92f07d3b3bba24a97e"
server_key_iv: "819a579e3cbfb15aed94916dd015178a"
server_key_version: "encryptionkey@0"
I20260926 20:43:55.599241 3934 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 20:43:55.600179 3949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:55.600383 3934 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:43:55.600487 3934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "29b11a4ac3b5434ca6d1ed95d0c38456"
format_stamp: "Formatted at 2026-09-26 20:43:55 on dist-test-slave-4hd3"
server_key: "75a1fc636e6d7b92f07d3b3bba24a97e"
server_key_iv: "819a579e3cbfb15aed94916dd015178a"
server_key_version: "encryptionkey@0"
I20260926 20:43:55.600556 3934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:55.620608 3934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:55.623701 3934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:55.623895 3934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:55.629328 3934 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:35799
I20260926 20:43:55.629364 4001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:35799 every 8 connection(s)
I20260926 20:43:55.629823 3934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:43:55.630726 4002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:55.633001 4002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456: Bootstrap starting.
I20260926 20:43:55.633723 4002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:55.633944 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 3934
I20260926 20:43:55.634051 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:43:55.634071 4002 log.cc:824] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:55.634236 393 external_mini_cluster.cc:1428] Setting key 5f8bd649444751b8da571111900e8354
I20260926 20:43:55.641400 4002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456: No bootstrap required, opened a new log
Sep 26 20:43:55 dist-test-slave-4hd3 krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:55.643147 4002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b11a4ac3b5434ca6d1ed95d0c38456" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35799 } }
I20260926 20:43:55.643313 4002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:55.643363 4002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b11a4ac3b5434ca6d1ed95d0c38456, State: Initialized, Role: FOLLOWER
I20260926 20:43:55.643498 4002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b11a4ac3b5434ca6d1ed95d0c38456" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35799 } }
I20260926 20:43:55.643595 4002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:43:55.643640 4002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:43:55.643692 4002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:55.644419 4002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b11a4ac3b5434ca6d1ed95d0c38456" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35799 } }
I20260926 20:43:55.644531 4002 leader_election.cc:302] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29b11a4ac3b5434ca6d1ed95d0c38456; no voters:
I20260926 20:43:55.644706 4002 leader_election.cc:288] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:43:55.644923 4007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:55.644961 4002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:43:55.645082 4007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456 [term 1 LEADER]: Becoming Leader. State: Replica: 29b11a4ac3b5434ca6d1ed95d0c38456, State: Running, Role: LEADER
I20260926 20:43:55.645164 4007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b11a4ac3b5434ca6d1ed95d0c38456" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35799 } }
I20260926 20:43:55.645567 4008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29b11a4ac3b5434ca6d1ed95d0c38456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b11a4ac3b5434ca6d1ed95d0c38456" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35799 } } }
I20260926 20:43:55.645641 4008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456 [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:55.645925 4016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:43:55.645958 4009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b11a4ac3b5434ca6d1ed95d0c38456. Latest consensus state: current_term: 1 leader_uuid: "29b11a4ac3b5434ca6d1ed95d0c38456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b11a4ac3b5434ca6d1ed95d0c38456" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35799 } } }
I20260926 20:43:55.646121 4009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456 [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:55.646448 4016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:43:55.646929 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:55.634878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.0.98.126:35799)
0926 20:43:55.635035 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:55.635044 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:55.635080 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:43:55.635479 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:55.635485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:55.635529 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:55.635692 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:55.636463 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:55.637816 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:55.638901 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:55.639493 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:55.639913 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:55.642145 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:55.642149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:55.642149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:55.642168 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:55.644796 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:55.645239 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:55.645242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:55.645242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:55.645355 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:55.645597 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:55.645598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:55.645599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:55.645908 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:55.646495 (+ 587us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:55.646574 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 20:43:55.646717 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260926 20:43:55.648352 4016 catalog_manager.cc:1347] Generated new cluster ID: 08077ffb69784762aa3c8c3766ecc3ec
I20260926 20:43:55.648427 4016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:43:55.658883 4016 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:43:55.659528 4016 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:43:55.671484 4016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29b11a4ac3b5434ca6d1ed95d0c38456: Generated new TSK 0
I20260926 20:43:55.671674 4016 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260926 20:43:55.698885 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35799
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:55.780223 4028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:55.780470 4028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:55.780494 4028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:55.781661 4028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:55.781701 4028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:55.781728 4028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:43:55.783288 4028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:55.783533 4028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:55.783788 4028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:55.788540 4034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:55.788540 4033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:55.788565 4036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:55.788805 4028 server_base.cc:1034] running on GCE node
I20260926 20:43:55.789124 4028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:55.789371 4028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:55.790510 4028 hybrid_clock.cc:648] HybridClock initialized: now 1790455435790501 us; error 26 us; skew 500 ppm
Sep 26 20:43:55 dist-test-slave-4hd3 krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:55.793004 4028 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:43:55.793431 4028 webserver.cc:460] Webserver started at http://127.0.98.65:40117/ using document root <none> and password file <none>
I20260926 20:43:55.793663 4028 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:55.793707 4028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:55.793851 4028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:55.794880 4028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "3680a98d38b646d180bb17e549bb6ec4"
format_stamp: "Formatted at 2026-09-26 20:43:55 on dist-test-slave-4hd3"
server_key: "3ffcecc0153aa613d2517991b7edfba8"
server_key_iv: "db48bf58c24e037b53c07bedef7e7664"
server_key_version: "encryptionkey@0"
I20260926 20:43:55.795241 4028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "3680a98d38b646d180bb17e549bb6ec4"
format_stamp: "Formatted at 2026-09-26 20:43:55 on dist-test-slave-4hd3"
server_key: "3ffcecc0153aa613d2517991b7edfba8"
server_key_iv: "db48bf58c24e037b53c07bedef7e7664"
server_key_version: "encryptionkey@0"
I20260926 20:43:55.796685 4028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:43:55.797469 4043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:55.797677 4028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:43:55.797765 4028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "3680a98d38b646d180bb17e549bb6ec4"
format_stamp: "Formatted at 2026-09-26 20:43:55 on dist-test-slave-4hd3"
server_key: "3ffcecc0153aa613d2517991b7edfba8"
server_key_iv: "db48bf58c24e037b53c07bedef7e7664"
server_key_version: "encryptionkey@0"
I20260926 20:43:55.797837 4028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:55.809742 4028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:55.811614 4028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:55.811779 4028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:55.811964 4028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:55.812266 4028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:55.812323 4028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:55.812363 4028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:55.812394 4028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:55.818053 4028 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:35739
I20260926 20:43:55.818109 4156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:35739 every 8 connection(s)
I20260926 20:43:55.818445 4028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:43:55.823423 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 4028
I20260926 20:43:55.823549 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:43:55.823802 393 external_mini_cluster.cc:1428] Setting key 15d6c6ea3f108c39f87b53bb9dc7d182
Sep 26 20:43:55 dist-test-slave-4hd3 krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:55.828359 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:55.819162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:43337 (local address 127.0.98.126:35799)
0926 20:43:55.819227 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:55.819229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:55.819453 (+ 224us) server_negotiation.cc:407] Connection header received
0926 20:43:55.820420 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:55.820421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:55.820433 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:55.820496 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:55.821112 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:55.821630 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:55.822641 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:55.822883 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:55.822970 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:55.825461 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:55.825462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:55.825462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:55.825471 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:55.826925 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:55.827357 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:55.827358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:55.827359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:55.827428 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:55.827685 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:55.827686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:55.827686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:55.827840 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:55.827920 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:55.828177 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 20:43:55.828268 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:43:55.828562 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:55.819152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35799 (local address 127.0.98.65:43337)
0926 20:43:55.819334 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:55.819367 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:55.820310 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:55.820572 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:55.820575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:55.820829 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:55.821010 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:55.821014 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:55.821726 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:55.821727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:55.822544 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:55.822546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:55.822979 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:55.822980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:55.823099 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:55.823691 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:55.823702 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:55.825370 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:55.826999 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:55.827004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:55.827006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:55.827257 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:55.827500 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:55.827500 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:55.827501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:55.827606 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:55.827909 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:55.827916 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:55.828045 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 20:43:55.828157 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 20:43:55.828320 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260926 20:43:55.829026 4157 heartbeater.cc:347] Connected to a master server at 127.0.98.126:35799
I20260926 20:43:55.829118 4157 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:55.829314 4157 heartbeater.cc:511] Master 127.0.98.126:35799 requested a full tablet report, sending...
I20260926 20:43:55.829833 3966 ts_manager.cc:194] Registered new tserver with Master: 3680a98d38b646d180bb17e549bb6ec4 (127.0.98.65:35739)
I20260926 20:43:55.830446 3966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:43337
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260926 20:43:55.850070 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35799
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:55.929545 4162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:55.929745 4162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:55.929768 4162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:55.930846 4162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:55.930881 4162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:55.930905 4162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:43:55.932296 4162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:55.932466 4162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:55.932626 4162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:55.937503 4170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:55.937495 4167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:55.937510 4168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:55.937642 4162 server_base.cc:1034] running on GCE node
I20260926 20:43:55.937958 4162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:55.938195 4162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:55.939334 4162 hybrid_clock.cc:648] HybridClock initialized: now 1790455435939337 us; error 42 us; skew 500 ppm
Sep 26 20:43:55 dist-test-slave-4hd3 krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:55.941902 4162 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:43:55.942365 4162 webserver.cc:460] Webserver started at http://127.0.98.66:41869/ using document root <none> and password file <none>
I20260926 20:43:55.942587 4162 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:55.942632 4162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:55.942791 4162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:55.943943 4162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "4b20109ae34e4eadb67fb60bc0fc96cf"
format_stamp: "Formatted at 2026-09-26 20:43:55 on dist-test-slave-4hd3"
server_key: "f13877067f9d138970ea26170987c3bb"
server_key_iv: "d1d1d2a2c9a3869654ed9d57f6442922"
server_key_version: "encryptionkey@0"
I20260926 20:43:55.944307 4162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "4b20109ae34e4eadb67fb60bc0fc96cf"
format_stamp: "Formatted at 2026-09-26 20:43:55 on dist-test-slave-4hd3"
server_key: "f13877067f9d138970ea26170987c3bb"
server_key_iv: "d1d1d2a2c9a3869654ed9d57f6442922"
server_key_version: "encryptionkey@0"
I20260926 20:43:55.945696 4162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:43:55.946470 4177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:55.946640 4162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:43:55.946736 4162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "4b20109ae34e4eadb67fb60bc0fc96cf"
format_stamp: "Formatted at 2026-09-26 20:43:55 on dist-test-slave-4hd3"
server_key: "f13877067f9d138970ea26170987c3bb"
server_key_iv: "d1d1d2a2c9a3869654ed9d57f6442922"
server_key_version: "encryptionkey@0"
I20260926 20:43:55.946817 4162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:55.970634 4162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:55.973067 4162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:55.973235 4162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:55.973507 4162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:55.973855 4162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:55.973892 4162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:55.973915 4162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:55.973927 4162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:55.980237 4162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:39825
I20260926 20:43:55.980316 4290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:39825 every 8 connection(s)
I20260926 20:43:55.980643 4162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:43:55.984802 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 4162
I20260926 20:43:55.984911 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:43:55.985100 393 external_mini_cluster.cc:1428] Setting key db125d2c55b739a35ac00c3d23ade991
Sep 26 20:43:55 dist-test-slave-4hd3 krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:55.991689 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:55.981359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:60277 (local address 127.0.98.126:35799)
0926 20:43:55.981404 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:55.981406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:55.981621 (+ 215us) server_negotiation.cc:407] Connection header received
0926 20:43:55.982650 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:55.982652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:55.982664 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:55.982718 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:55.983377 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:55.983953 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:55.985366 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:55.985560 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:55.985647 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:55.988493 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:55.988495 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:55.988495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:55.988504 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:55.990220 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:55.990637 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:55.990639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:55.990639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:55.990707 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:55.990955 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:55.990956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:55.990957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:55.991115 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:55.991194 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:55.991462 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 20:43:55.991555 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:43:55.991695 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:55.981340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35799 (local address 127.0.98.66:60277)
0926 20:43:55.981497 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:55.981532 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:55.982532 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:55.982803 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:55.982807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:55.983069 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:55.983269 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:55.983274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:55.984044 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:55.984045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:55.985265 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:55.985267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:55.985654 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:55.985655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:55.985821 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:55.986399 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:55.986410 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:55.988383 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:55.990309 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:55.990315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:55.990316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:55.990552 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:55.990790 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:55.990791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:55.990791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:55.990889 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:55.991209 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:55.991214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:55.991332 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 20:43:55.991412 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 20:43:55.991531 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260926 20:43:55.992218 4291 heartbeater.cc:347] Connected to a master server at 127.0.98.126:35799
I20260926 20:43:55.992334 4291 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:55.992532 4291 heartbeater.cc:511] Master 127.0.98.126:35799 requested a full tablet report, sending...
I20260926 20:43:55.992957 3966 ts_manager.cc:194] Registered new tserver with Master: 4b20109ae34e4eadb67fb60bc0fc96cf (127.0.98.66:39825)
I20260926 20:43:55.993347 3966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:60277
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260926 20:43:56.012256 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35799
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:56.093196 4296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:56.093396 4296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:56.093420 4296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:56.094542 4296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:56.094583 4296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:56.094609 4296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:43:56.096047 4296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:56.096230 4296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:56.096401 4296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:56.101361 4301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:56.101364 4302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:56.101361 4304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:56.101653 4296 server_base.cc:1034] running on GCE node
I20260926 20:43:56.101855 4296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:56.102056 4296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:56.103194 4296 hybrid_clock.cc:648] HybridClock initialized: now 1790455436103183 us; error 26 us; skew 500 ppm
Sep 26 20:43:56 dist-test-slave-4hd3 krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:56.105942 4296 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:43:56.106364 4296 webserver.cc:460] Webserver started at http://127.0.98.67:35389/ using document root <none> and password file <none>
I20260926 20:43:56.106604 4296 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:56.106675 4296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:56.106827 4296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:56.107928 4296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "15cbaaedc3494b9e9ed07b1b4fc58be4"
format_stamp: "Formatted at 2026-09-26 20:43:56 on dist-test-slave-4hd3"
server_key: "b05afe101c2f4659b2002390cf7d4f29"
server_key_iv: "c87087c24428270241116261e3d956de"
server_key_version: "encryptionkey@0"
I20260926 20:43:56.108273 4296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "15cbaaedc3494b9e9ed07b1b4fc58be4"
format_stamp: "Formatted at 2026-09-26 20:43:56 on dist-test-slave-4hd3"
server_key: "b05afe101c2f4659b2002390cf7d4f29"
server_key_iv: "c87087c24428270241116261e3d956de"
server_key_version: "encryptionkey@0"
I20260926 20:43:56.109537 4296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:43:56.110251 4311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:56.110423 4296 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:43:56.110527 4296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "15cbaaedc3494b9e9ed07b1b4fc58be4"
format_stamp: "Formatted at 2026-09-26 20:43:56 on dist-test-slave-4hd3"
server_key: "b05afe101c2f4659b2002390cf7d4f29"
server_key_iv: "c87087c24428270241116261e3d956de"
server_key_version: "encryptionkey@0"
I20260926 20:43:56.110599 4296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:56.122512 4296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:56.124369 4296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:56.124532 4296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:56.124774 4296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:56.125087 4296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:56.125119 4296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:56.125140 4296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:56.125154 4296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:56.130939 4296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:37705
I20260926 20:43:56.131003 4424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:37705 every 8 connection(s)
I20260926 20:43:56.131309 4296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:43:56.137048 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 4296
I20260926 20:43:56.137146 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
Sep 26 20:43:56 dist-test-slave-4hd3 krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:56.137327 393 external_mini_cluster.cc:1428] Setting key 9a70d43a36056c73982a09bae5576503
I20260926 20:43:56.140976 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:56.132057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:34403 (local address 127.0.98.126:35799)
0926 20:43:56.132105 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:56.132108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:56.132330 (+ 222us) server_negotiation.cc:407] Connection header received
0926 20:43:56.133216 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:56.133217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:56.133227 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:56.133276 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:56.133839 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:56.134319 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:56.135334 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:56.135517 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:56.135582 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:56.137990 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:56.137992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:56.137992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:56.138000 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:56.139542 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:56.139967 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:56.139968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:56.139968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:56.140018 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:56.140317 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:56.140318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:56.140319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:56.140437 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:56.140507 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:56.140799 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 20:43:56.140887 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:43:56.141188 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:56.132047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35799 (local address 127.0.98.67:34403)
0926 20:43:56.132201 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:56.132235 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:56.133127 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:56.133340 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:56.133343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:56.133585 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:56.133774 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:56.133777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:56.134392 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:56.134392 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:56.135168 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:56.135169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:56.135598 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:56.135598 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:56.135738 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:56.136265 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:56.136273 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:56.137862 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:56.139610 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:56.139615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:56.139617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:56.139913 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:56.140112 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:56.140113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:56.140113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:56.140239 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:56.140511 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:56.140517 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:56.140656 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 20:43:56.140779 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 20:43:56.140941 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260926 20:43:56.141625 4425 heartbeater.cc:347] Connected to a master server at 127.0.98.126:35799
I20260926 20:43:56.141709 4425 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:56.141896 4425 heartbeater.cc:511] Master 127.0.98.126:35799 requested a full tablet report, sending...
I20260926 20:43:56.142270 3966 ts_manager.cc:194] Registered new tserver with Master: 15cbaaedc3494b9e9ed07b1b4fc58be4 (127.0.98.67:37705)
I20260926 20:43:56.142668 3966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:34403
I20260926 20:43:56.149227 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:43:56 dist-test-slave-4hd3 krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:43:56 dist-test-slave-4hd3 krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260926 20:43:56.172644 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:56.164431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48932 (local address 127.0.98.65:35739)
0926 20:43:56.164591 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:56.164600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:56.164612 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:43:56.164639 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:56.164641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:56.164654 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:56.164723 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:56.165358 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:56.166080 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:56.166860 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:56.167177 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:56.167235 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:56.169554 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:56.169561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:56.169561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:56.169568 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:56.171253 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:56.171745 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:56.171749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:56.171749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:56.171854 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:56.172071 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:56.172072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:56.172073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:56.172223 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:56.172397 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:56.172445 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 20:43:56.172525 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
Sep 26 20:43:56 dist-test-slave-4hd3 krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:56.180787 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:56.173862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35084 (local address 127.0.98.126:35799)
0926 20:43:56.173905 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:56.173906 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:56.173932 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:43:56.173988 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:56.173989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:56.173997 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:56.174041 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:56.174541 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:56.175073 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:56.175822 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:56.176013 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:56.176063 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:56.178317 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:56.178319 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:56.178319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:56.178326 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:56.179756 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:56.180061 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:56.180061 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:56.180062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:56.180113 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:56.180303 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:56.180303 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:56.180304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:56.180449 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:56.180525 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:56.180666 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 20:43:56.180723 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 20:43:56.181083 3966 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35084
I20260926 20:43:56.181452 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 4028
I20260926 20:43:56.186546 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 4162
I20260926 20:43:56.190877 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 4296
I20260926 20:43:56.195258 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 3934
2026-09-26T20:43:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1859 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:43:56 dist-test-slave-4hd3 krb5kdc[4437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:43:56 dist-test-slave-4hd3 krb5kdc[4437](info): set up 2 sockets
Sep 26 20:43:56 dist-test-slave-4hd3 krb5kdc[4437](info): commencing operation
krb5kdc: starting...
W20260926 20:43:57.251871 393 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:43:57 dist-test-slave-4hd3 krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:43:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:43:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:43:57.363903 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33917
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:33917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:57.443567 4450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:57.443836 4450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:57.443868 4450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:57.445003 4450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:43:57.445037 4450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:57.445055 4450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:57.445066 4450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:43:57.445078 4450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:43:57.446537 4450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:33917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:57.446728 4450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:57.446894 4450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:57.451696 4458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:57.451788 4456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:57.451764 4455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:57.451975 4450 server_base.cc:1034] running on GCE node
I20260926 20:43:57.452253 4450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:57.452548 4450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:57.453706 4450 hybrid_clock.cc:648] HybridClock initialized: now 1790455437453688 us; error 48 us; skew 500 ppm
Sep 26 20:43:57 dist-test-slave-4hd3 krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:57.456172 4450 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:43:57.456547 4450 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:57.456616 4450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:57.456751 4450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:57.457851 4450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "18b9a2dbb7d640f6af691569253ee9b0"
format_stamp: "Formatted at 2026-09-26 20:43:57 on dist-test-slave-4hd3"
server_key: "472199ba6769e6050b27e26646946569"
server_key_iv: "28b5d0b30f13c4200b6c3b1e36af14ad"
server_key_version: "encryptionkey@0"
I20260926 20:43:57.458205 4450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "18b9a2dbb7d640f6af691569253ee9b0"
format_stamp: "Formatted at 2026-09-26 20:43:57 on dist-test-slave-4hd3"
server_key: "472199ba6769e6050b27e26646946569"
server_key_iv: "28b5d0b30f13c4200b6c3b1e36af14ad"
server_key_version: "encryptionkey@0"
I20260926 20:43:57.459499 4450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:43:57.460337 4464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:57.460529 4450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:43:57.460628 4450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "18b9a2dbb7d640f6af691569253ee9b0"
format_stamp: "Formatted at 2026-09-26 20:43:57 on dist-test-slave-4hd3"
server_key: "472199ba6769e6050b27e26646946569"
server_key_iv: "28b5d0b30f13c4200b6c3b1e36af14ad"
server_key_version: "encryptionkey@0"
I20260926 20:43:57.460714 4450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:57.481333 4450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:57.483263 4450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:57.483431 4450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:57.487658 4450 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:33917
I20260926 20:43:57.487702 4516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:33917 every 8 connection(s)
I20260926 20:43:57.488037 4450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:43:57.488386 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 4450
I20260926 20:43:57.488487 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:43:57.488673 393 external_mini_cluster.cc:1428] Setting key 6d0bb3904d43cc2f210dc84c6cbe4f43
I20260926 20:43:57.489200 4517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:57.492307 4517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0: Bootstrap starting.
I20260926 20:43:57.493028 4517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:57.493319 4517 log.cc:824] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:57.494050 4517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0: No bootstrap required, opened a new log
Sep 26 20:43:57 dist-test-slave-4hd3 krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:57.496284 4517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b9a2dbb7d640f6af691569253ee9b0" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33917 } }
I20260926 20:43:57.496495 4517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:57.496541 4517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b9a2dbb7d640f6af691569253ee9b0, State: Initialized, Role: FOLLOWER
I20260926 20:43:57.496701 4517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b9a2dbb7d640f6af691569253ee9b0" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33917 } }
I20260926 20:43:57.496785 4517 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:43:57.496809 4517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:43:57.496870 4517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:57.497493 4517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b9a2dbb7d640f6af691569253ee9b0" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33917 } }
I20260926 20:43:57.497584 4517 leader_election.cc:302] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18b9a2dbb7d640f6af691569253ee9b0; no voters:
I20260926 20:43:57.497762 4517 leader_election.cc:288] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:43:57.497804 4522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:57.497954 4522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0 [term 1 LEADER]: Becoming Leader. State: Replica: 18b9a2dbb7d640f6af691569253ee9b0, State: Running, Role: LEADER
I20260926 20:43:57.498019 4517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:43:57.498044 4522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b9a2dbb7d640f6af691569253ee9b0" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33917 } }
I20260926 20:43:57.498378 4523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18b9a2dbb7d640f6af691569253ee9b0. Latest consensus state: current_term: 1 leader_uuid: "18b9a2dbb7d640f6af691569253ee9b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b9a2dbb7d640f6af691569253ee9b0" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33917 } } }
I20260926 20:43:57.498476 4523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:57.498692 4528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:43:57.498972 4524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18b9a2dbb7d640f6af691569253ee9b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b9a2dbb7d640f6af691569253ee9b0" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33917 } } }
I20260926 20:43:57.499055 4524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:57.499223 4528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:43:57.499989 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:57.489202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45038 (local address 127.0.98.126:33917)
0926 20:43:57.489398 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:57.489406 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:57.489431 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:43:57.489851 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:57.489857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:57.489900 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:57.490013 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:57.490802 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:57.491708 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:57.492843 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:57.493304 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:57.493408 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:57.496101 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:57.496105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:57.496105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:57.496115 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:57.498175 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:57.498593 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:57.498596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:57.498597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:57.498715 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:57.498952 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:57.498953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:57.498954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:57.499292 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:57.499447 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:57.499661 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 20:43:57.499833 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
I20260926 20:43:57.501075 4539 mvcc.cc:205] Tried to move back new op lower bound from 7333705472001339392 to 7333705471992348672. Current Snapshot: MvccSnapshot[applied={T|T < 7333705472001339392}]
I20260926 20:43:57.501447 4528 catalog_manager.cc:1347] Generated new cluster ID: 0b4d7e4f76d4425fb9e1ba1aea3c08d3
I20260926 20:43:57.501506 4528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:43:57.511498 4528 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:43:57.512112 4528 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:43:57.515121 4528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0: Generated new TSK 0
I20260926 20:43:57.515326 4528 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260926 20:43:57.544267 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33917
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:57.626564 4543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:57.626762 4543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:57.626786 4543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:57.628012 4543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:57.628052 4543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:57.628080 4543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:43:57.629550 4543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:57.629787 4543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:57.630044 4543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:57.634958 4548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:57.635001 4543 server_base.cc:1034] running on GCE node
W20260926 20:43:57.634970 4549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:57.634970 4551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:57.635434 4543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:57.635747 4543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:57.636904 4543 hybrid_clock.cc:648] HybridClock initialized: now 1790455437636884 us; error 37 us; skew 500 ppm
Sep 26 20:43:57 dist-test-slave-4hd3 krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:57.639565 4543 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:43:57.639936 4543 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:57.639987 4543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:57.640126 4543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:57.641165 4543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "2f7777028ed04781b64357e52425e14f"
format_stamp: "Formatted at 2026-09-26 20:43:57 on dist-test-slave-4hd3"
server_key: "1973eafee2f3c458444d6b9b05526fc2"
server_key_iv: "05e4796abfbfd4ed6b7686bcd40651e6"
server_key_version: "encryptionkey@0"
I20260926 20:43:57.641541 4543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "2f7777028ed04781b64357e52425e14f"
format_stamp: "Formatted at 2026-09-26 20:43:57 on dist-test-slave-4hd3"
server_key: "1973eafee2f3c458444d6b9b05526fc2"
server_key_iv: "05e4796abfbfd4ed6b7686bcd40651e6"
server_key_version: "encryptionkey@0"
I20260926 20:43:57.642810 4543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:43:57.643487 4557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:57.643673 4543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:43:57.643776 4543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "2f7777028ed04781b64357e52425e14f"
format_stamp: "Formatted at 2026-09-26 20:43:57 on dist-test-slave-4hd3"
server_key: "1973eafee2f3c458444d6b9b05526fc2"
server_key_iv: "05e4796abfbfd4ed6b7686bcd40651e6"
server_key_version: "encryptionkey@0"
I20260926 20:43:57.643843 4543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:57.656612 4543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:57.658874 4543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:57.658981 4543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:57.659211 4543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:57.659487 4543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:57.659519 4543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:57.659543 4543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:57.659556 4543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:57.665495 4543 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:44907
I20260926 20:43:57.665562 4670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:44907 every 8 connection(s)
I20260926 20:43:57.665841 4543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:43:57.669412 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 4543
I20260926 20:43:57.669551 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:43:57.669725 393 external_mini_cluster.cc:1428] Setting key 3359c0d4c8d9ee726e6741b12f7845e8
Sep 26 20:43:57 dist-test-slave-4hd3 krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:57.676167 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:57.666528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33917 (local address 127.0.98.65:33153)
0926 20:43:57.666710 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:57.666777 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:57.667228 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:57.667430 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:57.667433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:57.667675 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:57.668241 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:57.668246 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:57.668947 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:57.668948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:57.669903 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:57.669905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:57.670416 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:57.670418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:57.670561 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:57.671117 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:57.671125 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:57.672846 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:57.674726 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:57.674732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:57.674734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:57.674967 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:57.675200 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:57.675200 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:57.675201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:57.675300 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:57.675589 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:57.675593 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:57.675758 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 20:43:57.675887 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 20:43:57.676008 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
I20260926 20:43:57.676167 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:57.666561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:33153 (local address 127.0.98.126:33917)
0926 20:43:57.666632 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:57.666634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:57.666862 (+ 228us) server_negotiation.cc:407] Connection header received
0926 20:43:57.667310 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:57.667311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:57.667321 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:57.667369 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:57.668343 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:57.668860 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:57.669997 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:57.670275 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:57.670347 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:57.672942 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:57.672944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:57.672944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:57.672952 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:57.674648 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:57.675046 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:57.675048 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:57.675048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:57.675127 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:57.675366 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:57.675367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:57.675367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:57.675520 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:57.675598 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:57.675933 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 20:43:57.676031 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:43:57.676878 4671 heartbeater.cc:347] Connected to a master server at 127.0.98.126:33917
I20260926 20:43:57.676981 4671 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:57.677146 4671 heartbeater.cc:511] Master 127.0.98.126:33917 requested a full tablet report, sending...
I20260926 20:43:57.677673 4481 ts_manager.cc:194] Registered new tserver with Master: 2f7777028ed04781b64357e52425e14f (127.0.98.65:44907)
I20260926 20:43:57.678303 4481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:33153
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260926 20:43:57.694193 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33917
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:57.773659 4676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:57.773886 4676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:57.773926 4676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:57.775063 4676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:57.775127 4676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:57.775174 4676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:43:57.776693 4676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:57.776966 4676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:57.777194 4676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:57.782016 4684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:57.781988 4681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:57.782094 4682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:57.782297 4676 server_base.cc:1034] running on GCE node
I20260926 20:43:57.782531 4676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:57.782796 4676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:57.783941 4676 hybrid_clock.cc:648] HybridClock initialized: now 1790455437783918 us; error 37 us; skew 500 ppm
Sep 26 20:43:57 dist-test-slave-4hd3 krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:57.786602 4676 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:43:57.786994 4676 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:57.787046 4676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:57.787216 4676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:57.788326 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "411541d410d6492d968304111c918db2"
format_stamp: "Formatted at 2026-09-26 20:43:57 on dist-test-slave-4hd3"
server_key: "94c65b6e0fddf91d730e773c9e06cdb3"
server_key_iv: "e6fb0cc1319282206aeafcb710484f93"
server_key_version: "encryptionkey@0"
I20260926 20:43:57.788676 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "411541d410d6492d968304111c918db2"
format_stamp: "Formatted at 2026-09-26 20:43:57 on dist-test-slave-4hd3"
server_key: "94c65b6e0fddf91d730e773c9e06cdb3"
server_key_iv: "e6fb0cc1319282206aeafcb710484f93"
server_key_version: "encryptionkey@0"
I20260926 20:43:57.789968 4676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:43:57.790679 4690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:57.790865 4676 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:43:57.790961 4676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "411541d410d6492d968304111c918db2"
format_stamp: "Formatted at 2026-09-26 20:43:57 on dist-test-slave-4hd3"
server_key: "94c65b6e0fddf91d730e773c9e06cdb3"
server_key_iv: "e6fb0cc1319282206aeafcb710484f93"
server_key_version: "encryptionkey@0"
I20260926 20:43:57.791042 4676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:57.813472 4676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:57.815434 4676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:57.815604 4676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:57.815889 4676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:57.816253 4676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:57.816316 4676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:57.816357 4676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:57.816381 4676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:57.822041 4676 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:44919
I20260926 20:43:57.822090 4803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:44919 every 8 connection(s)
I20260926 20:43:57.822356 4676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:43:57 dist-test-slave-4hd3 krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:57.828781 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 4676
I20260926 20:43:57.828897 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:43:57.829079 393 external_mini_cluster.cc:1428] Setting key beec714425f7d33759245d16b42ce799
I20260926 20:43:57.831686 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:57.823041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:48349 (local address 127.0.98.126:33917)
0926 20:43:57.823108 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:57.823110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:57.823300 (+ 190us) server_negotiation.cc:407] Connection header received
0926 20:43:57.823876 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:57.823877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:57.823889 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:57.823944 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:57.824863 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:57.825353 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:57.826360 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:57.826548 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:57.826652 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:57.828674 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:57.828675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:57.828676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:57.828683 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:57.830197 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:57.830622 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:57.830623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:57.830624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:57.830684 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:57.830990 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:57.830992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:57.830992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:57.831147 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:57.831206 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:57.831563 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 20:43:57.831624 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:43:57.831990 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:57.823013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33917 (local address 127.0.98.66:48349)
0926 20:43:57.823173 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:57.823217 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:57.823760 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:57.824016 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:57.824020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:57.824278 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:57.824756 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:57.824760 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:57.825453 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:57.825454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:57.826249 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:57.826251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:57.826658 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:57.826659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:57.826787 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:57.827234 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:57.827242 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:57.828555 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:57.830300 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:57.830307 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:57.830310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:57.830530 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:57.830752 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:57.830752 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:57.830753 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:57.830905 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:57.831217 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:57.831223 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:57.831415 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 20:43:57.831532 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 20:43:57.831702 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260926 20:43:57.832497 4804 heartbeater.cc:347] Connected to a master server at 127.0.98.126:33917
I20260926 20:43:57.832594 4804 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:57.832746 4804 heartbeater.cc:511] Master 127.0.98.126:33917 requested a full tablet report, sending...
I20260926 20:43:57.833097 4481 ts_manager.cc:194] Registered new tserver with Master: 411541d410d6492d968304111c918db2 (127.0.98.66:44919)
I20260926 20:43:57.833467 4481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:48349
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260926 20:43:57.854187 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33917
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:43:57.934568 4809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:57.934763 4809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:57.934787 4809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:57.935997 4809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:57.936038 4809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:57.936065 4809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:43:57.937556 4809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:57.937777 4809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:57.938009 4809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:57.942929 4814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:57.942970 4815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:57.942979 4817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:57.943059 4809 server_base.cc:1034] running on GCE node
I20260926 20:43:57.943336 4809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:57.943575 4809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:57.944748 4809 hybrid_clock.cc:648] HybridClock initialized: now 1790455437944728 us; error 36 us; skew 500 ppm
Sep 26 20:43:57 dist-test-slave-4hd3 krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:43:57.947527 4809 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:43:57.947956 4809 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:57.948031 4809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:57.948194 4809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:57.949290 4809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "9904e28247ff47459882df623c10105f"
format_stamp: "Formatted at 2026-09-26 20:43:57 on dist-test-slave-4hd3"
server_key: "e961ce885d24bf964d12db0d37108e6a"
server_key_iv: "8390d2ef2112f9d115e73a27f40186f2"
server_key_version: "encryptionkey@0"
I20260926 20:43:57.949648 4809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "9904e28247ff47459882df623c10105f"
format_stamp: "Formatted at 2026-09-26 20:43:57 on dist-test-slave-4hd3"
server_key: "e961ce885d24bf964d12db0d37108e6a"
server_key_iv: "8390d2ef2112f9d115e73a27f40186f2"
server_key_version: "encryptionkey@0"
I20260926 20:43:57.950979 4809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:43:57.951825 4823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:57.952001 4809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:43:57.952098 4809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "9904e28247ff47459882df623c10105f"
format_stamp: "Formatted at 2026-09-26 20:43:57 on dist-test-slave-4hd3"
server_key: "e961ce885d24bf964d12db0d37108e6a"
server_key_iv: "8390d2ef2112f9d115e73a27f40186f2"
server_key_version: "encryptionkey@0"
I20260926 20:43:57.952180 4809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:57.968439 4809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:57.970678 4809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:57.970912 4809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:57.971165 4809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:57.971491 4809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:57.971547 4809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:57.971593 4809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:57.971625 4809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:57.977559 4809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:41201
I20260926 20:43:57.977612 4936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:41201 every 8 connection(s)
I20260926 20:43:57.977921 4809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:43:57.979053 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 4809
I20260926 20:43:57.979159 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:43:57.979383 393 external_mini_cluster.cc:1428] Setting key c34be4a2770e95bc6738f1271d3aa440
Sep 26 20:43:57 dist-test-slave-4hd3 krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:43:57.987522 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:57.978699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:35733 (local address 127.0.98.126:33917)
0926 20:43:57.978768 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:57.978770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:57.979075 (+ 305us) server_negotiation.cc:407] Connection header received
0926 20:43:57.979872 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:57.979873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:57.979885 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:57.979934 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:57.980897 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:57.981398 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:57.982393 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:57.982604 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:57.982689 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:57.984937 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:57.984938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:57.984939 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:57.984945 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:57.986266 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:57.986662 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:57.986663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:57.986663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:57.986713 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:57.986967 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:57.986968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:57.986968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:57.987066 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:57.987116 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:57.987394 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 20:43:57.987460 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:43:57.987600 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:57.978664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33917 (local address 127.0.98.67:35733)
0926 20:43:57.978912 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:57.978962 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:57.979669 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:57.980004 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:57.980008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:57.980277 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:57.980790 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:57.980795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:57.981504 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:57.981505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:57.982306 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:57.982308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:57.982698 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:57.982699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:57.982829 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:57.983307 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:43:57.983316 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:57.984838 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:57.986339 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:57.986345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:57.986347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:57.986588 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:57.986805 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:43:57.986806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:43:57.986806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:43:57.986912 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:43:57.987123 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:57.987128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:57.987239 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 20:43:57.987329 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:43:57.987445 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":146,"threads_started":1}
I20260926 20:43:57.988169 4937 heartbeater.cc:347] Connected to a master server at 127.0.98.126:33917
I20260926 20:43:57.988248 4937 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:57.988438 4937 heartbeater.cc:511] Master 127.0.98.126:33917 requested a full tablet report, sending...
I20260926 20:43:57.988843 4481 ts_manager.cc:194] Registered new tserver with Master: 9904e28247ff47459882df623c10105f (127.0.98.67:41201)
I20260926 20:43:57.989202 4481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:35733
I20260926 20:43:57.991003 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:43:57.997953 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:57.992343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45052 (local address 127.0.98.126:33917)
0926 20:43:57.992409 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:57.992411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:57.992500 (+ 89us) server_negotiation.cc:407] Connection header received
0926 20:43:57.992533 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:57.992534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:57.992542 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:57.992585 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:57.993109 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:57.993559 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:57.994368 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:57.994540 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:57.994594 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:57.995812 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:57.995813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:57.995813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:43:57.995819 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:57.996990 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:57.997316 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:57.997317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:57.997317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:57.997368 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:43:57.997535 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:43:57.997535 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:43:57.997535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:43:57.997656 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:57.997736 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:57.997806 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 20:43:57.997861 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:43:57.999644 4481 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:43:58.001391 4481 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:43:58.007678 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.003566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:44907 (local address 127.0.0.1:50552)
0926 20:43:58.004040 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:58.004048 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:58.004074 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:58.004414 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:58.004415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:58.004417 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:58.004457 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:58.004458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.005314 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.005315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:58.006513 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:58.006515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.007466 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.007467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:58.007523 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:58.007526 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:43:58.007574 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 20:43:58.007607 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":6,"thread_start_us":54,"threads_started":1}
I20260926 20:43:58.007773 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.003751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50552 (local address 127.0.98.65:44907)
0926 20:43:58.004251 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:58.004256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:58.004275 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:43:58.004313 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:58.004315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:58.004330 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:58.004403 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:58.004515 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.005202 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.006596 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.007354 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.007462 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:58.007520 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:58.007601 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 20:43:58.007651 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":78,"threads_started":1}
I20260926 20:43:58.007942 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.003698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.0.98.67:41201)
0926 20:43:58.003967 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:58.003972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:58.003980 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:43:58.004021 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:58.004022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:58.004034 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:58.004120 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:58.004580 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.005638 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.006749 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.007254 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.007341 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:58.007400 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:58.007646 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 20:43:58.007685 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":123,"threads_started":1}
I20260926 20:43:58.007953 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.003559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:44919 (local address 127.0.0.1:59414)
0926 20:43:58.003904 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:58.003916 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:58.003969 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:58.004305 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:58.004306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:58.004309 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:58.004358 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:58.004360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.005674 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.005675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:58.006756 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:58.006758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.007708 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.007709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:58.007797 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:58.007801 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:43:58.007836 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 20:43:58.007881 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":3,"thread_start_us":45,"threads_started":1}
I20260926 20:43:58.008389 4605 tablet_service.cc:1467] Processing CreateTablet for tablet 779316e9078a44639a61ad57263d3259 (DEFAULT_TABLE table=test-table [id=0eef7ad4a6324b2895bf62536a670f41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:58.008705 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.003838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.0.98.66:44919)
0926 20:43:58.004155 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:58.004159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:58.004166 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:43:58.004196 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:58.004198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:58.004210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:58.004294 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:58.004439 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.005567 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.006836 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.007598 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.008454 (+ 856us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:58.008543 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:58.008570 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 20:43:58.008618 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":64,"threads_started":1}
I20260926 20:43:58.008791 4605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779316e9078a44639a61ad57263d3259. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:58.009349 4738 tablet_service.cc:1467] Processing CreateTablet for tablet 779316e9078a44639a61ad57263d3259 (DEFAULT_TABLE table=test-table [id=0eef7ad4a6324b2895bf62536a670f41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:58.009662 4738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779316e9078a44639a61ad57263d3259. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:58.011830 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.003546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:41201 (local address 127.0.0.1:40852)
0926 20:43:58.003696 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:58.003813 (+ 117us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:58.003879 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:58.004123 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:58.004126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:58.004128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:58.004514 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:58.004516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.005721 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.005722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:58.006641 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:58.006643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.007339 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.007340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:58.007419 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:58.007432 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 20:43:58.007498 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 20:43:58.007553 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":49,"threads_started":1}
I20260926 20:43:58.012214 4951 tablet_bootstrap.cc:492] T 779316e9078a44639a61ad57263d3259 P 2f7777028ed04781b64357e52425e14f: Bootstrap starting.
I20260926 20:43:58.012842 4951 tablet_bootstrap.cc:654] T 779316e9078a44639a61ad57263d3259 P 2f7777028ed04781b64357e52425e14f: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:58.012907 4952 tablet_bootstrap.cc:492] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2: Bootstrap starting.
I20260926 20:43:58.013389 4951 log.cc:824] T 779316e9078a44639a61ad57263d3259 P 2f7777028ed04781b64357e52425e14f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:58.013514 4952 tablet_bootstrap.cc:654] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:58.013631 4871 tablet_service.cc:1467] Processing CreateTablet for tablet 779316e9078a44639a61ad57263d3259 (DEFAULT_TABLE table=test-table [id=0eef7ad4a6324b2895bf62536a670f41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:58.013976 4871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779316e9078a44639a61ad57263d3259. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:58.014113 4952 log.cc:824] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:58.014846 4952 tablet_bootstrap.cc:492] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2: No bootstrap required, opened a new log
I20260926 20:43:58.014912 4952 ts_tablet_manager.cc:1397] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:43:58.017010 4952 raft_consensus.cc:348] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411541d410d6492d968304111c918db2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44919 } } peers { permanent_uuid: "9904e28247ff47459882df623c10105f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41201 } } peers { permanent_uuid: "2f7777028ed04781b64357e52425e14f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44907 } }
I20260926 20:43:58.017163 4955 tablet_bootstrap.cc:492] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f: Bootstrap starting.
I20260926 20:43:58.017199 4952 raft_consensus.cc:374] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:58.017246 4952 raft_consensus.cc:729] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 411541d410d6492d968304111c918db2, State: Initialized, Role: FOLLOWER
I20260926 20:43:58.017382 4952 consensus_queue.cc:260] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411541d410d6492d968304111c918db2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44919 } } peers { permanent_uuid: "9904e28247ff47459882df623c10105f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41201 } } peers { permanent_uuid: "2f7777028ed04781b64357e52425e14f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44907 } }
I20260926 20:43:58.017681 4952 ts_tablet_manager.cc:1428] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:43:58.017812 4804 heartbeater.cc:503] Master 127.0.98.126:33917 was elected leader, sending a full tablet report...
I20260926 20:43:58.017824 4955 tablet_bootstrap.cc:654] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:58.017861 4951 tablet_bootstrap.cc:492] T 779316e9078a44639a61ad57263d3259 P 2f7777028ed04781b64357e52425e14f: No bootstrap required, opened a new log
I20260926 20:43:58.017920 4951 ts_tablet_manager.cc:1397] T 779316e9078a44639a61ad57263d3259 P 2f7777028ed04781b64357e52425e14f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 20:43:58.018400 4955 log.cc:824] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:58.019348 4955 tablet_bootstrap.cc:492] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f: No bootstrap required, opened a new log
I20260926 20:43:58.019426 4955 ts_tablet_manager.cc:1397] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:43:58.019642 4951 raft_consensus.cc:348] T 779316e9078a44639a61ad57263d3259 P 2f7777028ed04781b64357e52425e14f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411541d410d6492d968304111c918db2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44919 } } peers { permanent_uuid: "9904e28247ff47459882df623c10105f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41201 } } peers { permanent_uuid: "2f7777028ed04781b64357e52425e14f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44907 } }
I20260926 20:43:58.019835 4951 raft_consensus.cc:374] T 779316e9078a44639a61ad57263d3259 P 2f7777028ed04781b64357e52425e14f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:58.019877 4951 raft_consensus.cc:729] T 779316e9078a44639a61ad57263d3259 P 2f7777028ed04781b64357e52425e14f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f7777028ed04781b64357e52425e14f, State: Initialized, Role: FOLLOWER
I20260926 20:43:58.019954 4951 consensus_queue.cc:260] T 779316e9078a44639a61ad57263d3259 P 2f7777028ed04781b64357e52425e14f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411541d410d6492d968304111c918db2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44919 } } peers { permanent_uuid: "9904e28247ff47459882df623c10105f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41201 } } peers { permanent_uuid: "2f7777028ed04781b64357e52425e14f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44907 } }
I20260926 20:43:58.020141 4951 ts_tablet_manager.cc:1428] T 779316e9078a44639a61ad57263d3259 P 2f7777028ed04781b64357e52425e14f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:43:58.020179 4671 heartbeater.cc:503] Master 127.0.98.126:33917 was elected leader, sending a full tablet report...
I20260926 20:43:58.021376 4955 raft_consensus.cc:348] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411541d410d6492d968304111c918db2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44919 } } peers { permanent_uuid: "9904e28247ff47459882df623c10105f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41201 } } peers { permanent_uuid: "2f7777028ed04781b64357e52425e14f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44907 } }
I20260926 20:43:58.021515 4955 raft_consensus.cc:374] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:58.021559 4955 raft_consensus.cc:729] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9904e28247ff47459882df623c10105f, State: Initialized, Role: FOLLOWER
I20260926 20:43:58.021668 4955 consensus_queue.cc:260] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411541d410d6492d968304111c918db2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44919 } } peers { permanent_uuid: "9904e28247ff47459882df623c10105f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41201 } } peers { permanent_uuid: "2f7777028ed04781b64357e52425e14f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44907 } }
I20260926 20:43:58.021921 4955 ts_tablet_manager.cc:1428] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:43:58.021962 4937 heartbeater.cc:503] Master 127.0.98.126:33917 was elected leader, sending a full tablet report...
I20260926 20:43:58.033423 4956 raft_consensus.cc:482] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:43:58.033519 4956 raft_consensus.cc:504] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411541d410d6492d968304111c918db2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44919 } } peers { permanent_uuid: "9904e28247ff47459882df623c10105f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41201 } } peers { permanent_uuid: "2f7777028ed04781b64357e52425e14f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44907 } }
I20260926 20:43:58.033936 4956 leader_election.cc:288] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9904e28247ff47459882df623c10105f (127.0.98.67:41201), 2f7777028ed04781b64357e52425e14f (127.0.98.65:44907)
I20260926 20:43:58.036795 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.034173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:40105 (local address 127.0.98.67:41201)
0926 20:43:58.034225 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:58.034228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:58.034281 (+ 53us) server_negotiation.cc:407] Connection header received
0926 20:43:58.034373 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:58.034374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:58.034386 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:58.034443 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:58.034585 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.035049 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.036067 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.036434 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.036535 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:58.036545 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:58.036697 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 20:43:58.036732 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 20:43:58.036792 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.034081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:41201 (local address 127.0.98.66:40105)
0926 20:43:58.034203 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:58.034215 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:58.034293 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:58.034446 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:58.034447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:58.034448 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:58.034515 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:58.034517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.035127 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.035128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:58.035974 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:58.035976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.036549 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.036550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:58.036619 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:58.036622 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:43:58.036680 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:43:58.036721 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260926 20:43:58.036876 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.034199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:42657 (local address 127.0.98.65:44907)
0926 20:43:58.034243 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:58.034245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:58.034373 (+ 128us) server_negotiation.cc:407] Connection header received
0926 20:43:58.034452 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:58.034453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:58.034463 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:58.034527 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:43:58.034680 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.035149 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.036147 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.036525 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.036608 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:58.036616 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:58.036776 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 20:43:58.036811 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:43:58.036964 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.034135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:44907 (local address 127.0.98.66:42657)
0926 20:43:58.034294 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:58.034304 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:58.034388 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:58.034542 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:58.034543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:58.034545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:43:58.034615 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:58.034617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.035231 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.035232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:58.036063 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:58.036065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.036608 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.036609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:58.036684 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:58.036687 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:43:58.036767 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 20:43:58.036804 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260926 20:43:58.037241 4625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779316e9078a44639a61ad57263d3259" candidate_uuid: "411541d410d6492d968304111c918db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f7777028ed04781b64357e52425e14f" is_pre_election: true
I20260926 20:43:58.037247 4891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779316e9078a44639a61ad57263d3259" candidate_uuid: "411541d410d6492d968304111c918db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9904e28247ff47459882df623c10105f" is_pre_election: true
I20260926 20:43:58.037402 4891 raft_consensus.cc:2454] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411541d410d6492d968304111c918db2 in term 0.
I20260926 20:43:58.037400 4625 raft_consensus.cc:2454] T 779316e9078a44639a61ad57263d3259 P 2f7777028ed04781b64357e52425e14f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411541d410d6492d968304111c918db2 in term 0.
I20260926 20:43:58.037617 4692 leader_election.cc:302] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f7777028ed04781b64357e52425e14f, 411541d410d6492d968304111c918db2; no voters:
I20260926 20:43:58.037755 4956 raft_consensus.cc:2781] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:43:58.037832 4956 raft_consensus.cc:482] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:43:58.037868 4956 raft_consensus.cc:3037] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:58.038667 4956 raft_consensus.cc:504] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411541d410d6492d968304111c918db2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44919 } } peers { permanent_uuid: "9904e28247ff47459882df623c10105f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41201 } } peers { permanent_uuid: "2f7777028ed04781b64357e52425e14f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44907 } }
I20260926 20:43:58.038810 4956 leader_election.cc:288] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [CANDIDATE]: Term 1 election: Requested vote from peers 9904e28247ff47459882df623c10105f (127.0.98.67:41201), 2f7777028ed04781b64357e52425e14f (127.0.98.65:44907)
I20260926 20:43:58.038938 4625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779316e9078a44639a61ad57263d3259" candidate_uuid: "411541d410d6492d968304111c918db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f7777028ed04781b64357e52425e14f"
I20260926 20:43:58.038944 4891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779316e9078a44639a61ad57263d3259" candidate_uuid: "411541d410d6492d968304111c918db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9904e28247ff47459882df623c10105f"
I20260926 20:43:58.039017 4891 raft_consensus.cc:3037] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:58.039034 4625 raft_consensus.cc:3037] T 779316e9078a44639a61ad57263d3259 P 2f7777028ed04781b64357e52425e14f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:58.039703 4891 raft_consensus.cc:2454] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411541d410d6492d968304111c918db2 in term 1.
I20260926 20:43:58.039705 4625 raft_consensus.cc:2454] T 779316e9078a44639a61ad57263d3259 P 2f7777028ed04781b64357e52425e14f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411541d410d6492d968304111c918db2 in term 1.
I20260926 20:43:58.039906 4692 leader_election.cc:302] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f7777028ed04781b64357e52425e14f, 411541d410d6492d968304111c918db2; no voters:
I20260926 20:43:58.040000 4956 raft_consensus.cc:2781] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:58.040133 4956 raft_consensus.cc:686] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [term 1 LEADER]: Becoming Leader. State: Replica: 411541d410d6492d968304111c918db2, State: Running, Role: LEADER
I20260926 20:43:58.040226 4956 consensus_queue.cc:237] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411541d410d6492d968304111c918db2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44919 } } peers { permanent_uuid: "9904e28247ff47459882df623c10105f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41201 } } peers { permanent_uuid: "2f7777028ed04781b64357e52425e14f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44907 } }
I20260926 20:43:58.041030 4481 catalog_manager.cc:5495] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 411541d410d6492d968304111c918db2 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "411541d410d6492d968304111c918db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411541d410d6492d968304111c918db2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9904e28247ff47459882df623c10105f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f7777028ed04781b64357e52425e14f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44907 } health_report { overall_health: UNKNOWN } } }
I20260926 20:43:58.052736 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.050224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.0.98.66:44919)
0926 20:43:58.050276 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:58.050279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:58.050304 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:43:58.050417 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:58.050418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:58.050438 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:58.050522 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:43:58.050651 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.051123 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.051932 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.052139 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.052222 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:58.052395 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:43:58.052501 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:43:58.052595 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:58.052615 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 20:43:58.052633 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 20:43:58.054649 4625 raft_consensus.cc:1261] T 779316e9078a44639a61ad57263d3259 P 2f7777028ed04781b64357e52425e14f [term 1 FOLLOWER]: Refusing update from remote peer 411541d410d6492d968304111c918db2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:58.054649 4891 raft_consensus.cc:1261] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f [term 1 FOLLOWER]: Refusing update from remote peer 411541d410d6492d968304111c918db2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:58.055016 4961 consensus_queue.cc:1035] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9904e28247ff47459882df623c10105f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:43:58.055146 4956 consensus_queue.cc:1035] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f7777028ed04781b64357e52425e14f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:43:58.057329 4964 mvcc.cc:205] Tried to move back new op lower bound from 7333705474268872704 to 7333705474213486592. Current Snapshot: MvccSnapshot[applied={T|T < 7333705474268872704}]
I20260926 20:43:58.058260 4965 mvcc.cc:205] Tried to move back new op lower bound from 7333705474268872704 to 7333705474213486592. Current Snapshot: MvccSnapshot[applied={T|T < 7333705474268872704}]
I20260926 20:43:58.058349 4966 mvcc.cc:205] Tried to move back new op lower bound from 7333705474268872704 to 7333705474213486592. Current Snapshot: MvccSnapshot[applied={T|T < 7333705474268872704 or (T in {7333705474268872704})}]
I20260926 20:43:58.059549 4481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:43:58.059641 4481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:43:58.060454 4481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 18b9a2dbb7d640f6af691569253ee9b0: Sending DeleteTablet for 3 replicas of tablet 779316e9078a44639a61ad57263d3259
I20260926 20:43:58.060793 4871 tablet_service.cc:1514] Processing DeleteTablet for tablet 779316e9078a44639a61ad57263d3259 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:58 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:40852
I20260926 20:43:58.060793 4605 tablet_service.cc:1514] Processing DeleteTablet for tablet 779316e9078a44639a61ad57263d3259 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:58 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:50552
I20260926 20:43:58.061029 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 4543
I20260926 20:43:58.061137 4738 tablet_service.cc:1514] Processing DeleteTablet for tablet 779316e9078a44639a61ad57263d3259 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:58 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:59414
I20260926 20:43:58.061141 4974 tablet_replica.cc:330] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f: stopping tablet replica
I20260926 20:43:58.063392 4974 raft_consensus.cc:2229] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:43:58.063520 4974 raft_consensus.cc:2258] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:58.063935 4974 ts_tablet_manager.cc:1905] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:43:58.065526 4974 ts_tablet_manager.cc:1918] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:43:58.065591 4974 log.cc:1196] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/779316e9078a44639a61ad57263d3259
I20260926 20:43:58.065876 4974 ts_tablet_manager.cc:1939] T 779316e9078a44639a61ad57263d3259 P 9904e28247ff47459882df623c10105f: Deleting consensus metadata
I20260926 20:43:58.066308 4467 catalog_manager.cc:4841] TS 9904e28247ff47459882df623c10105f (127.0.98.67:41201): tablet 779316e9078a44639a61ad57263d3259 (table test-table [id=0eef7ad4a6324b2895bf62536a670f41]) successfully deleted
I20260926 20:43:58.067863 4975 tablet_replica.cc:330] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2: stopping tablet replica
I20260926 20:43:58.067974 4975 raft_consensus.cc:2229] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:43:58.068128 4975 raft_consensus.cc:2258] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:58.068583 4975 ts_tablet_manager.cc:1905] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:43:58.070225 4975 ts_tablet_manager.cc:1918] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:43:58.070307 4975 log.cc:1196] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/779316e9078a44639a61ad57263d3259
I20260926 20:43:58.070572 4975 ts_tablet_manager.cc:1939] T 779316e9078a44639a61ad57263d3259 P 411541d410d6492d968304111c918db2: Deleting consensus metadata
I20260926 20:43:58.071026 4465 catalog_manager.cc:4841] TS 411541d410d6492d968304111c918db2 (127.0.98.66:44919): tablet 779316e9078a44639a61ad57263d3259 (table test-table [id=0eef7ad4a6324b2895bf62536a670f41]) successfully deleted
W20260926 20:43:58.071520 4466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:44907 (error 108)
I20260926 20:43:58.071797 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 4676
I20260926 20:43:58.077508 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 4809
I20260926 20:43:58.077674 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.077515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:44907 (local address 127.0.0.1:50568)
0926 20:43:58.077588 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:58.077632 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 20:43:58.079411 4466 catalog_manager.cc:4584] TS 2f7777028ed04781b64357e52425e14f (127.0.98.65:44907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 779316e9078a44639a61ad57263d3259: Network error: Client connection negotiation failed: client connection to 127.0.98.65:44907: connect: Connection refused (error 111)
I20260926 20:43:58.082549 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 4450
2026-09-26T20:43:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1887 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T20:43:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:43:58Z Disabled control of system clock
I20260926 20:43:58.109937 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34365
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:34365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 20:43:58.189421 4980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:58.189620 4980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:58.189641 4980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:58.190786 4980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:43:58.190827 4980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:58.190840 4980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:58.190876 4980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:43:58.190903 4980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:43:58.192296 4980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:34365
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:58.192535 4980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:58.192833 4980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:58.197690 4988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:58.197705 4986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:58.197921 4980 server_base.cc:1034] running on GCE node
W20260926 20:43:58.197966 4985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:58.198271 4980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:58.198516 4980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:58.199667 4980 hybrid_clock.cc:648] HybridClock initialized: now 1790455438199650 us; error 38 us; skew 500 ppm
I20260926 20:43:58.201012 4980 webserver.cc:460] Webserver started at http://127.0.98.126:33659/ using document root <none> and password file <none>
I20260926 20:43:58.201261 4980 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:58.201336 4980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:58.201508 4980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:58.202533 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "8a9a4548c1ae44c68a34ce6c8d9f7b54"
format_stamp: "Formatted at 2026-09-26 20:43:58 on dist-test-slave-4hd3"
server_key: "cabacf0529b83b4e2f300b4cd7c45403"
server_key_iv: "5094796d3d164bd282f27fd192476e9b"
server_key_version: "encryptionkey@0"
I20260926 20:43:58.202889 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "8a9a4548c1ae44c68a34ce6c8d9f7b54"
format_stamp: "Formatted at 2026-09-26 20:43:58 on dist-test-slave-4hd3"
server_key: "cabacf0529b83b4e2f300b4cd7c45403"
server_key_iv: "5094796d3d164bd282f27fd192476e9b"
server_key_version: "encryptionkey@0"
I20260926 20:43:58.204296 4980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:43:58.205010 4994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:58.205179 4980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:43:58.205302 4980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "8a9a4548c1ae44c68a34ce6c8d9f7b54"
format_stamp: "Formatted at 2026-09-26 20:43:58 on dist-test-slave-4hd3"
server_key: "cabacf0529b83b4e2f300b4cd7c45403"
server_key_iv: "5094796d3d164bd282f27fd192476e9b"
server_key_version: "encryptionkey@0"
I20260926 20:43:58.205379 4980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:58.208645 4980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:58.208880 4980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:58.209028 4980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:58.212711 4980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:34365
I20260926 20:43:58.212747 5046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:34365 every 8 connection(s)
I20260926 20:43:58.213125 4980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:43:58.213774 5047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:58.213987 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 4980
I20260926 20:43:58.214083 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:43:58.214247 393 external_mini_cluster.cc:1428] Setting key e090e52f03921164051a2166fdee7e29
I20260926 20:43:58.216806 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.214781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56124 (local address 127.0.98.126:34365)
0926 20:43:58.214981 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:58.214991 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:58.215023 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:43:58.215494 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:58.215502 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:58.215552 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:58.215753 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:58.216392 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:58.216396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:58.216396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:43:58.216400 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:58.216443 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:43:58.216456 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:58.216533 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:58.216570 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 20:43:58.216624 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260926 20:43:58.217628 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54: Bootstrap starting.
I20260926 20:43:58.218185 5047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:58.218461 5047 log.cc:824] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:58.219094 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54: No bootstrap required, opened a new log
I20260926 20:43:58.220417 5047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a4548c1ae44c68a34ce6c8d9f7b54" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34365 } }
I20260926 20:43:58.220538 5047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:58.220561 5047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9a4548c1ae44c68a34ce6c8d9f7b54, State: Initialized, Role: FOLLOWER
I20260926 20:43:58.220701 5047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a4548c1ae44c68a34ce6c8d9f7b54" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34365 } }
I20260926 20:43:58.220781 5047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:43:58.220826 5047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:43:58.220873 5047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:58.221376 5047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a4548c1ae44c68a34ce6c8d9f7b54" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34365 } }
I20260926 20:43:58.221455 5047 leader_election.cc:302] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a9a4548c1ae44c68a34ce6c8d9f7b54; no voters:
I20260926 20:43:58.221642 5047 leader_election.cc:288] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:43:58.221897 5047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:43:58.221993 5052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:58.222144 5052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54 [term 1 LEADER]: Becoming Leader. State: Replica: 8a9a4548c1ae44c68a34ce6c8d9f7b54, State: Running, Role: LEADER
I20260926 20:43:58.222249 5052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a4548c1ae44c68a34ce6c8d9f7b54" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34365 } }
I20260926 20:43:58.222641 5053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a9a4548c1ae44c68a34ce6c8d9f7b54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a4548c1ae44c68a34ce6c8d9f7b54" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34365 } } }
I20260926 20:43:58.222644 5054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a9a4548c1ae44c68a34ce6c8d9f7b54. Latest consensus state: current_term: 1 leader_uuid: "8a9a4548c1ae44c68a34ce6c8d9f7b54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a4548c1ae44c68a34ce6c8d9f7b54" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34365 } } }
I20260926 20:43:58.222748 5053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54 [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:58.222761 5054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54 [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:58.222930 5061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:43:58.223387 5061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:43:58.224864 5061 catalog_manager.cc:1347] Generated new cluster ID: 2dfe4688d7384e3a8444a4264988303a
I20260926 20:43:58.224908 5061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:43:58.266346 5061 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:43:58.266542 5061 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:43:58.269346 5061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54: Generated new TSK 0
I20260926 20:43:58.269541 5061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:43:58.279503 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34365
--builtin_ntp_servers=127.0.98.84:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 20:43:58.361580 5071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:58.361783 5071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:58.361804 5071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:58.362947 5071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:58.362985 5071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:58.363039 5071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:43:58.364473 5071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:58.364718 5071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:58.365057 5071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:58.369932 5077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:58.370010 5079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:58.369901 5076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:58.370056 5071 server_base.cc:1034] running on GCE node
I20260926 20:43:58.370273 5071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:58.370530 5071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:58.371696 5071 hybrid_clock.cc:648] HybridClock initialized: now 1790455438371682 us; error 24 us; skew 500 ppm
I20260926 20:43:58.372983 5071 webserver.cc:460] Webserver started at http://127.0.98.65:45881/ using document root <none> and password file <none>
I20260926 20:43:58.373220 5071 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:58.373313 5071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:58.373466 5071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:58.374559 5071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "e62d95f3018d45c9be0c0acdb2dd09ba"
format_stamp: "Formatted at 2026-09-26 20:43:58 on dist-test-slave-4hd3"
server_key: "bb68832019c90bb96605c12cec4551af"
server_key_iv: "2a1aa9a98f168f5a28246cdad0197c0a"
server_key_version: "encryptionkey@0"
I20260926 20:43:58.374920 5071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "e62d95f3018d45c9be0c0acdb2dd09ba"
format_stamp: "Formatted at 2026-09-26 20:43:58 on dist-test-slave-4hd3"
server_key: "bb68832019c90bb96605c12cec4551af"
server_key_iv: "2a1aa9a98f168f5a28246cdad0197c0a"
server_key_version: "encryptionkey@0"
I20260926 20:43:58.376336 5071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 20:43:58.377031 5085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:58.377204 5071 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:43:58.377310 5071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "e62d95f3018d45c9be0c0acdb2dd09ba"
format_stamp: "Formatted at 2026-09-26 20:43:58 on dist-test-slave-4hd3"
server_key: "bb68832019c90bb96605c12cec4551af"
server_key_iv: "2a1aa9a98f168f5a28246cdad0197c0a"
server_key_version: "encryptionkey@0"
I20260926 20:43:58.377382 5071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:58.380674 5071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:58.380932 5071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:58.381084 5071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:58.381289 5071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:58.381572 5071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:58.381630 5071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:58.381667 5071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:58.381695 5071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:58.387369 5071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:40259
I20260926 20:43:58.387410 5198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:40259 every 8 connection(s)
I20260926 20:43:58.387827 5071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:43:58.390604 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.388552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:42519 (local address 127.0.98.126:34365)
0926 20:43:58.388619 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:58.388621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:58.388864 (+ 243us) server_negotiation.cc:407] Connection header received
0926 20:43:58.389304 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:58.389305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:58.389309 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:58.389365 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:58.390423 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:58.390423 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:58.390424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:43:58.390427 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:58.390438 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:43:58.390441 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:58.390485 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:58.390540 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 20:43:58.390553 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:43:58.390775 5201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.388541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34365 (local address 127.0.98.65:42519)
0926 20:43:58.388724 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:58.388764 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:58.389221 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:58.389423 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:58.389426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:58.389434 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:58.390299 (+ 865us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:43:58.390300 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:58.390326 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:43:58.390352 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:58.390482 (+ 130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:58.390485 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:58.390487 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:43:58.390536 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 20:43:58.390580 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260926 20:43:58.391347 5199 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34365
I20260926 20:43:58.391449 5199 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:58.391590 5199 heartbeater.cc:511] Master 127.0.98.126:34365 requested a full tablet report, sending...
I20260926 20:43:58.392124 5011 ts_manager.cc:194] Registered new tserver with Master: e62d95f3018d45c9be0c0acdb2dd09ba (127.0.98.65:40259)
I20260926 20:43:58.394256 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 5071
I20260926 20:43:58.394371 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:43:58.394559 393 external_mini_cluster.cc:1428] Setting key 9142a90a33e321934c2feb06c66f7b85
I20260926 20:43:58.395437 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34365
--builtin_ntp_servers=127.0.98.84:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 20:43:58.475068 5202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:58.475310 5202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:58.475333 5202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:58.476568 5202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:58.476609 5202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:58.476634 5202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:43:58.477968 5202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:58.478217 5202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:58.478511 5202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:58.483613 5207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:58.483637 5210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:58.483637 5208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:58.483865 5202 server_base.cc:1034] running on GCE node
I20260926 20:43:58.484040 5202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:58.484237 5202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:58.485394 5202 hybrid_clock.cc:648] HybridClock initialized: now 1790455438485383 us; error 28 us; skew 500 ppm
I20260926 20:43:58.486629 5202 webserver.cc:460] Webserver started at http://127.0.98.66:38033/ using document root <none> and password file <none>
I20260926 20:43:58.486879 5202 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:58.486974 5202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:58.487118 5202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:58.488293 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "d474d5513d734e92998087f391f67e41"
format_stamp: "Formatted at 2026-09-26 20:43:58 on dist-test-slave-4hd3"
server_key: "5c5ed7fd9d955bcc6c62df954a24f59f"
server_key_iv: "880c95d2dc2eef10735f7cab4f1633d4"
server_key_version: "encryptionkey@0"
I20260926 20:43:58.488641 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "d474d5513d734e92998087f391f67e41"
format_stamp: "Formatted at 2026-09-26 20:43:58 on dist-test-slave-4hd3"
server_key: "5c5ed7fd9d955bcc6c62df954a24f59f"
server_key_iv: "880c95d2dc2eef10735f7cab4f1633d4"
server_key_version: "encryptionkey@0"
I20260926 20:43:58.489936 5202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:43:58.490685 5216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:58.490899 5202 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:43:58.490996 5202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "d474d5513d734e92998087f391f67e41"
format_stamp: "Formatted at 2026-09-26 20:43:58 on dist-test-slave-4hd3"
server_key: "5c5ed7fd9d955bcc6c62df954a24f59f"
server_key_iv: "880c95d2dc2eef10735f7cab4f1633d4"
server_key_version: "encryptionkey@0"
I20260926 20:43:58.491079 5202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:58.494385 5202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:58.494669 5202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:58.494822 5202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:58.495020 5202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:58.495328 5202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:58.495380 5202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:58.495417 5202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:58.495445 5202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:58.501142 5202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:45361
I20260926 20:43:58.501188 5329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:45361 every 8 connection(s)
I20260926 20:43:58.501528 5202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:43:58.504364 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.502273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:34265 (local address 127.0.98.126:34365)
0926 20:43:58.502331 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:58.502334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:58.502565 (+ 231us) server_negotiation.cc:407] Connection header received
0926 20:43:58.502986 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:58.502986 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:58.502991 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:58.503048 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:58.504166 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:58.504168 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:58.504168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:43:58.504171 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:58.504184 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:43:58.504188 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:58.504233 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:58.504300 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 20:43:58.504312 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:43:58.504551 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.502255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34365 (local address 127.0.98.66:34265)
0926 20:43:58.502436 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:58.502474 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:58.502907 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:58.503105 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:58.503108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:58.503116 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:58.504037 (+ 921us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:43:58.504038 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:58.504065 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:43:58.504090 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:58.504233 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:58.504236 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:58.504238 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:43:58.504295 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 20:43:58.504338 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260926 20:43:58.505082 5330 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34365
I20260926 20:43:58.505156 5330 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:58.505270 5330 heartbeater.cc:511] Master 127.0.98.126:34365 requested a full tablet report, sending...
I20260926 20:43:58.505576 5011 ts_manager.cc:194] Registered new tserver with Master: d474d5513d734e92998087f391f67e41 (127.0.98.66:45361)
I20260926 20:43:58.509778 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 5202
I20260926 20:43:58.509889 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:43:58.510074 393 external_mini_cluster.cc:1428] Setting key 7674fdd7b7bf71e64648f5bf600edfb5
I20260926 20:43:58.510919 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34365
--builtin_ntp_servers=127.0.98.84:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 20:43:58.591009 5333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:58.591228 5333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:58.591249 5333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:58.592478 5333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:58.592523 5333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:58.592549 5333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:43:58.593999 5333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:58.594239 5333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:58.594583 5333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:58.599587 5339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:58.599587 5341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:58.599612 5338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:58.599900 5333 server_base.cc:1034] running on GCE node
I20260926 20:43:58.600102 5333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:58.600307 5333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:58.601480 5333 hybrid_clock.cc:648] HybridClock initialized: now 1790455438601470 us; error 38 us; skew 500 ppm
I20260926 20:43:58.602857 5333 webserver.cc:460] Webserver started at http://127.0.98.67:34153/ using document root <none> and password file <none>
I20260926 20:43:58.603103 5333 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:58.603175 5333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:58.603339 5333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:58.604472 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "be9cb4898b1b449bb6f66c82b9186898"
format_stamp: "Formatted at 2026-09-26 20:43:58 on dist-test-slave-4hd3"
server_key: "921e9999705a886e14c95ff80280aa30"
server_key_iv: "667fb035ad268243a37f49d6f65f27a1"
server_key_version: "encryptionkey@0"
I20260926 20:43:58.604847 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "be9cb4898b1b449bb6f66c82b9186898"
format_stamp: "Formatted at 2026-09-26 20:43:58 on dist-test-slave-4hd3"
server_key: "921e9999705a886e14c95ff80280aa30"
server_key_iv: "667fb035ad268243a37f49d6f65f27a1"
server_key_version: "encryptionkey@0"
I20260926 20:43:58.606199 5333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:43:58.606927 5347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:58.607107 5333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:43:58.607206 5333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "be9cb4898b1b449bb6f66c82b9186898"
format_stamp: "Formatted at 2026-09-26 20:43:58 on dist-test-slave-4hd3"
server_key: "921e9999705a886e14c95ff80280aa30"
server_key_iv: "667fb035ad268243a37f49d6f65f27a1"
server_key_version: "encryptionkey@0"
I20260926 20:43:58.607282 5333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:58.610782 5333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:58.611038 5333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:58.611177 5333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:58.611379 5333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:43:58.611665 5333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:43:58.611742 5333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:58.611783 5333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:43:58.611809 5333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:58.617588 5333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:42271
I20260926 20:43:58.617624 5460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:42271 every 8 connection(s)
I20260926 20:43:58.617961 5333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:43:58.621213 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.618666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:59035 (local address 127.0.98.126:34365)
0926 20:43:58.618786 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:58.618788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:58.618980 (+ 192us) server_negotiation.cc:407] Connection header received
0926 20:43:58.619673 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:58.619674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:58.619681 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:58.619784 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:58.620979 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:58.620980 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:58.620981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:43:58.620986 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:58.621001 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:43:58.621005 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:58.621057 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:58.621130 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 20:43:58.621145 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 20:43:58.621490 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.618667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34365 (local address 127.0.98.67:59035)
0926 20:43:58.618859 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:58.618898 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:58.619562 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:58.619855 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:58.619859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:58.619869 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:58.620829 (+ 960us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:43:58.620829 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:58.620856 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:43:58.620883 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:58.621054 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:58.621058 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:58.621062 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:43:58.621139 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:43:58.621253 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260926 20:43:58.622010 5461 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34365
I20260926 20:43:58.622117 5461 heartbeater.cc:464] Registering TS with master...
I20260926 20:43:58.622256 5461 heartbeater.cc:511] Master 127.0.98.126:34365 requested a full tablet report, sending...
I20260926 20:43:58.622685 5011 ts_manager.cc:194] Registered new tserver with Master: be9cb4898b1b449bb6f66c82b9186898 (127.0.98.67:42271)
I20260926 20:43:58.625512 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 5333
I20260926 20:43:58.625622 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:43:58.625846 393 external_mini_cluster.cc:1428] Setting key b834b3b35a70a2443ee375d228aa801a
I20260926 20:43:58.626678 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:43:58.628990 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.628033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56126 (local address 127.0.98.126:34365)
0926 20:43:58.628090 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:58.628091 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:58.628175 (+ 84us) server_negotiation.cc:407] Connection header received
0926 20:43:58.628238 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:58.628238 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:58.628243 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:58.628282 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:58.628854 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:58.628855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:58.628855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:43:58.628857 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:58.628866 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:43:58.628869 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:58.628906 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:58.628936 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 20:43:58.628946 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:43:58.630614 5011 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:43:58.632266 5011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:43:58.635957 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.634275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:42271 (local address 127.0.0.1:45380)
0926 20:43:58.634391 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:58.634428 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:58.634478 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:58.634993 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:58.634999 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:58.635005 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:58.635459 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:43:58.635460 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:58.635476 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:43:58.635500 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:58.635783 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:58.635789 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:58.635791 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:43:58.635853 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 20:43:58.635867 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260926 20:43:58.636159 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.634435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45380 (local address 127.0.98.67:42271)
0926 20:43:58.634844 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:58.634848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:58.634857 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:43:58.634879 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:58.634881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:58.634898 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:58.634978 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:58.635583 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:58.635590 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:58.635590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:43:58.635594 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:58.635630 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:43:58.635655 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:58.635755 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:58.636063 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 20:43:58.636080 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":83,"threads_started":1}
I20260926 20:43:58.636264 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.634446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:45361 (local address 127.0.0.1:58944)
0926 20:43:58.634605 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:58.634615 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:58.634649 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:58.635021 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:58.635022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:58.635026 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:58.635730 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:43:58.635730 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:58.635737 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:43:58.635750 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:58.636131 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:58.636132 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:58.636133 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:43:58.636161 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 20:43:58.636173 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":69,"threads_started":1}
I20260926 20:43:58.636446 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.634487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:40259 (local address 127.0.0.1:42360)
0926 20:43:58.634659 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:58.634670 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:58.634718 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:58.635567 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:58.635568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:58.635570 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:58.635950 (+ 380us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:43:58.635950 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:58.635956 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:43:58.635966 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:58.636292 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:58.636293 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:58.636293 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 20:43:58.636354 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 20:43:58.636367 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":79,"threads_started":1}
I20260926 20:43:58.636583 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.634632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58944 (local address 127.0.98.66:45361)
0926 20:43:58.634864 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:58.634868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:58.634879 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:43:58.634904 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:58.634906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:58.634915 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:58.635495 (+ 580us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:58.635967 (+ 472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:58.635970 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:58.635970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:43:58.635973 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:58.636027 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:43:58.636044 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:58.636441 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:58.636464 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 20:43:58.636479 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":135,"threads_started":1}
I20260926 20:43:58.636636 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.634697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42360 (local address 127.0.98.65:40259)
0926 20:43:58.635410 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:58.635415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:58.635425 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:43:58.635450 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:58.635452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:58.635460 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:58.635767 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:58.636170 (+ 403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:58.636173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:58.636173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:43:58.636177 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:58.636216 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:43:58.636231 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:58.636298 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:58.636544 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 20:43:58.636563 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":119,"threads_started":1}
I20260926 20:43:58.636987 5395 tablet_service.cc:1467] Processing CreateTablet for tablet bc4a3b4dbddd442097cf75f68aa9c429 (DEFAULT_TABLE table=test-table [id=980458b1d2a44cf8bff0c25eb8db9f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:58.637171 5264 tablet_service.cc:1467] Processing CreateTablet for tablet bc4a3b4dbddd442097cf75f68aa9c429 (DEFAULT_TABLE table=test-table [id=980458b1d2a44cf8bff0c25eb8db9f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:58.637355 5395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc4a3b4dbddd442097cf75f68aa9c429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:58.637521 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc4a3b4dbddd442097cf75f68aa9c429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:58.640105 5475 tablet_bootstrap.cc:492] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41: Bootstrap starting.
I20260926 20:43:58.640133 5476 tablet_bootstrap.cc:492] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898: Bootstrap starting.
I20260926 20:43:58.640785 5476 tablet_bootstrap.cc:654] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:58.640817 5475 tablet_bootstrap.cc:654] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:58.640890 5133 tablet_service.cc:1467] Processing CreateTablet for tablet bc4a3b4dbddd442097cf75f68aa9c429 (DEFAULT_TABLE table=test-table [id=980458b1d2a44cf8bff0c25eb8db9f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:43:58.641129 5476 log.cc:824] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:58.641140 5475 log.cc:824] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:58.641256 5133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc4a3b4dbddd442097cf75f68aa9c429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:58.641885 5476 tablet_bootstrap.cc:492] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898: No bootstrap required, opened a new log
I20260926 20:43:58.641955 5476 ts_tablet_manager.cc:1397] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:43:58.641993 5475 tablet_bootstrap.cc:492] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41: No bootstrap required, opened a new log
I20260926 20:43:58.642066 5475 ts_tablet_manager.cc:1397] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:43:58.644018 5475 raft_consensus.cc:348] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9cb4898b1b449bb6f66c82b9186898" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42271 } } peers { permanent_uuid: "d474d5513d734e92998087f391f67e41" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45361 } } peers { permanent_uuid: "e62d95f3018d45c9be0c0acdb2dd09ba" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40259 } }
I20260926 20:43:58.644169 5475 raft_consensus.cc:374] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:58.644212 5475 raft_consensus.cc:729] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d474d5513d734e92998087f391f67e41, State: Initialized, Role: FOLLOWER
I20260926 20:43:58.644351 5475 consensus_queue.cc:260] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9cb4898b1b449bb6f66c82b9186898" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42271 } } peers { permanent_uuid: "d474d5513d734e92998087f391f67e41" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45361 } } peers { permanent_uuid: "e62d95f3018d45c9be0c0acdb2dd09ba" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40259 } }
I20260926 20:43:58.644634 5475 ts_tablet_manager.cc:1428] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:43:58.644716 5479 tablet_bootstrap.cc:492] T bc4a3b4dbddd442097cf75f68aa9c429 P e62d95f3018d45c9be0c0acdb2dd09ba: Bootstrap starting.
I20260926 20:43:58.644749 5330 heartbeater.cc:503] Master 127.0.98.126:34365 was elected leader, sending a full tablet report...
I20260926 20:43:58.644680 5476 raft_consensus.cc:348] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9cb4898b1b449bb6f66c82b9186898" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42271 } } peers { permanent_uuid: "d474d5513d734e92998087f391f67e41" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45361 } } peers { permanent_uuid: "e62d95f3018d45c9be0c0acdb2dd09ba" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40259 } }
I20260926 20:43:58.644852 5476 raft_consensus.cc:374] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:58.644887 5476 raft_consensus.cc:729] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be9cb4898b1b449bb6f66c82b9186898, State: Initialized, Role: FOLLOWER
I20260926 20:43:58.645008 5476 consensus_queue.cc:260] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9cb4898b1b449bb6f66c82b9186898" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42271 } } peers { permanent_uuid: "d474d5513d734e92998087f391f67e41" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45361 } } peers { permanent_uuid: "e62d95f3018d45c9be0c0acdb2dd09ba" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40259 } }
I20260926 20:43:58.645243 5476 ts_tablet_manager.cc:1428] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 20:43:58.645350 5461 heartbeater.cc:503] Master 127.0.98.126:34365 was elected leader, sending a full tablet report...
I20260926 20:43:58.645650 5479 tablet_bootstrap.cc:654] T bc4a3b4dbddd442097cf75f68aa9c429 P e62d95f3018d45c9be0c0acdb2dd09ba: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:58.645972 5479 log.cc:824] T bc4a3b4dbddd442097cf75f68aa9c429 P e62d95f3018d45c9be0c0acdb2dd09ba: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:58.646647 5479 tablet_bootstrap.cc:492] T bc4a3b4dbddd442097cf75f68aa9c429 P e62d95f3018d45c9be0c0acdb2dd09ba: No bootstrap required, opened a new log
I20260926 20:43:58.646730 5479 ts_tablet_manager.cc:1397] T bc4a3b4dbddd442097cf75f68aa9c429 P e62d95f3018d45c9be0c0acdb2dd09ba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:43:58.648559 5479 raft_consensus.cc:348] T bc4a3b4dbddd442097cf75f68aa9c429 P e62d95f3018d45c9be0c0acdb2dd09ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9cb4898b1b449bb6f66c82b9186898" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42271 } } peers { permanent_uuid: "d474d5513d734e92998087f391f67e41" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45361 } } peers { permanent_uuid: "e62d95f3018d45c9be0c0acdb2dd09ba" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40259 } }
I20260926 20:43:58.648702 5479 raft_consensus.cc:374] T bc4a3b4dbddd442097cf75f68aa9c429 P e62d95f3018d45c9be0c0acdb2dd09ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:58.648748 5479 raft_consensus.cc:729] T bc4a3b4dbddd442097cf75f68aa9c429 P e62d95f3018d45c9be0c0acdb2dd09ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e62d95f3018d45c9be0c0acdb2dd09ba, State: Initialized, Role: FOLLOWER
I20260926 20:43:58.648861 5479 consensus_queue.cc:260] T bc4a3b4dbddd442097cf75f68aa9c429 P e62d95f3018d45c9be0c0acdb2dd09ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9cb4898b1b449bb6f66c82b9186898" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42271 } } peers { permanent_uuid: "d474d5513d734e92998087f391f67e41" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45361 } } peers { permanent_uuid: "e62d95f3018d45c9be0c0acdb2dd09ba" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40259 } }
I20260926 20:43:58.649111 5479 ts_tablet_manager.cc:1428] T bc4a3b4dbddd442097cf75f68aa9c429 P e62d95f3018d45c9be0c0acdb2dd09ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 20:43:58.649291 5199 heartbeater.cc:503] Master 127.0.98.126:34365 was elected leader, sending a full tablet report...
I20260926 20:43:58.736943 5480 raft_consensus.cc:482] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:43:58.737100 5480 raft_consensus.cc:504] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9cb4898b1b449bb6f66c82b9186898" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42271 } } peers { permanent_uuid: "d474d5513d734e92998087f391f67e41" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45361 } } peers { permanent_uuid: "e62d95f3018d45c9be0c0acdb2dd09ba" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40259 } }
I20260926 20:43:58.737488 5480 leader_election.cc:288] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be9cb4898b1b449bb6f66c82b9186898 (127.0.98.67:42271), e62d95f3018d45c9be0c0acdb2dd09ba (127.0.98.65:40259)
I20260926 20:43:58.738875 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.737768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:40019 (local address 127.0.98.67:42271)
0926 20:43:58.737846 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:58.737849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:58.737861 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:43:58.737895 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:58.737896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:58.737902 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:58.737962 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:58.738647 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:58.738649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:58.738649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:43:58.738654 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:58.738674 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:43:58.738678 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:58.738741 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:58.738795 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 20:43:58.738808 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:43:58.738905 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.737695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:42271 (local address 127.0.98.66:40019)
0926 20:43:58.737769 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:58.737783 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:58.737849 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:58.737999 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:58.738000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:58.738003 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:58.738549 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:43:58.738549 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:58.738557 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:43:58.738567 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:58.738737 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:58.738738 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:58.738739 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:43:58.738793 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:43:58.738808 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 20:43:58.739131 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.737827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:40259 (local address 127.0.98.66:59069)
0926 20:43:58.738003 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:58.738012 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:58.738053 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:58.738237 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:58.738237 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:58.738240 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:43:58.738762 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:43:58.738762 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:43:58.738769 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:43:58.738780 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:43:58.738985 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:43:58.738986 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:43:58.738987 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:43:58.739043 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 20:43:58.739057 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
I20260926 20:43:58.739151 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.738038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:59069 (local address 127.0.98.65:40259)
0926 20:43:58.738109 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:58.738111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:58.738121 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:43:58.738145 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:58.738146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:58.738150 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:58.738211 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:58.738875 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:58.738876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:58.738877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:43:58.738881 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:58.738896 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:43:58.738901 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:58.738955 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:58.739071 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 20:43:58.739087 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:43:58.739187 5415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4a3b4dbddd442097cf75f68aa9c429" candidate_uuid: "d474d5513d734e92998087f391f67e41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9cb4898b1b449bb6f66c82b9186898" is_pre_election: true
I20260926 20:43:58.739339 5415 raft_consensus.cc:2454] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d474d5513d734e92998087f391f67e41 in term 0.
I20260926 20:43:58.739535 5153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4a3b4dbddd442097cf75f68aa9c429" candidate_uuid: "d474d5513d734e92998087f391f67e41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e62d95f3018d45c9be0c0acdb2dd09ba" is_pre_election: true
I20260926 20:43:58.739581 5218 leader_election.cc:302] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be9cb4898b1b449bb6f66c82b9186898, d474d5513d734e92998087f391f67e41; no voters:
I20260926 20:43:58.739653 5153 raft_consensus.cc:2454] T bc4a3b4dbddd442097cf75f68aa9c429 P e62d95f3018d45c9be0c0acdb2dd09ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d474d5513d734e92998087f391f67e41 in term 0.
I20260926 20:43:58.739699 5480 raft_consensus.cc:2781] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:43:58.739799 5480 raft_consensus.cc:482] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:43:58.739847 5480 raft_consensus.cc:3037] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:58.740536 5480 raft_consensus.cc:504] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9cb4898b1b449bb6f66c82b9186898" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42271 } } peers { permanent_uuid: "d474d5513d734e92998087f391f67e41" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45361 } } peers { permanent_uuid: "e62d95f3018d45c9be0c0acdb2dd09ba" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40259 } }
I20260926 20:43:58.740666 5480 leader_election.cc:288] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [CANDIDATE]: Term 1 election: Requested vote from peers be9cb4898b1b449bb6f66c82b9186898 (127.0.98.67:42271), e62d95f3018d45c9be0c0acdb2dd09ba (127.0.98.65:40259)
I20260926 20:43:58.740813 5415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4a3b4dbddd442097cf75f68aa9c429" candidate_uuid: "d474d5513d734e92998087f391f67e41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9cb4898b1b449bb6f66c82b9186898"
I20260926 20:43:58.740819 5153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4a3b4dbddd442097cf75f68aa9c429" candidate_uuid: "d474d5513d734e92998087f391f67e41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e62d95f3018d45c9be0c0acdb2dd09ba"
I20260926 20:43:58.740893 5415 raft_consensus.cc:3037] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:58.740909 5153 raft_consensus.cc:3037] T bc4a3b4dbddd442097cf75f68aa9c429 P e62d95f3018d45c9be0c0acdb2dd09ba [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:58.741745 5153 raft_consensus.cc:2454] T bc4a3b4dbddd442097cf75f68aa9c429 P e62d95f3018d45c9be0c0acdb2dd09ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d474d5513d734e92998087f391f67e41 in term 1.
I20260926 20:43:58.741745 5415 raft_consensus.cc:2454] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d474d5513d734e92998087f391f67e41 in term 1.
I20260926 20:43:58.741922 5218 leader_election.cc:302] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d474d5513d734e92998087f391f67e41, e62d95f3018d45c9be0c0acdb2dd09ba; no voters:
I20260926 20:43:58.742020 5480 raft_consensus.cc:2781] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:58.742182 5480 raft_consensus.cc:686] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [term 1 LEADER]: Becoming Leader. State: Replica: d474d5513d734e92998087f391f67e41, State: Running, Role: LEADER
I20260926 20:43:58.742293 5480 consensus_queue.cc:237] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9cb4898b1b449bb6f66c82b9186898" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42271 } } peers { permanent_uuid: "d474d5513d734e92998087f391f67e41" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45361 } } peers { permanent_uuid: "e62d95f3018d45c9be0c0acdb2dd09ba" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40259 } }
I20260926 20:43:58.743117 5011 catalog_manager.cc:5495] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 reported cstate change: term changed from 0 to 1, leader changed from <none> to d474d5513d734e92998087f391f67e41 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "d474d5513d734e92998087f391f67e41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9cb4898b1b449bb6f66c82b9186898" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d474d5513d734e92998087f391f67e41" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e62d95f3018d45c9be0c0acdb2dd09ba" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40259 } health_report { overall_health: UNKNOWN } } }
I20260926 20:43:58.751668 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.750660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.0.98.66:45361)
0926 20:43:58.750721 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:58.750723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:58.750732 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:43:58.750788 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:58.750789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:58.750801 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:58.750892 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:58.751419 (+ 527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:58.751420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:58.751421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:43:58.751425 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:58.751440 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:43:58.751444 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:58.751500 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:58.751580 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 20:43:58.751594 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 20:43:58.752022 5331 tablet.cc:2369] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:43:58.753614 5415 raft_consensus.cc:1261] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898 [term 1 FOLLOWER]: Refusing update from remote peer d474d5513d734e92998087f391f67e41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:58.753614 5153 raft_consensus.cc:1261] T bc4a3b4dbddd442097cf75f68aa9c429 P e62d95f3018d45c9be0c0acdb2dd09ba [term 1 FOLLOWER]: Refusing update from remote peer d474d5513d734e92998087f391f67e41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:43:58.753985 5480 consensus_queue.cc:1035] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e62d95f3018d45c9be0c0acdb2dd09ba" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:43:58.754129 5485 consensus_queue.cc:1035] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be9cb4898b1b449bb6f66c82b9186898" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:43:58.756799 5490 mvcc.cc:205] Tried to move back new op lower bound from 7333705477131776000 to 7333705477089144832. Current Snapshot: MvccSnapshot[applied={T|T < 7333705477131776000}]
I20260926 20:43:58.758774 5010 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:43:58.758854 5010 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:43:58.759608 5010 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a9a4548c1ae44c68a34ce6c8d9f7b54: Sending DeleteTablet for 3 replicas of tablet bc4a3b4dbddd442097cf75f68aa9c429
I20260926 20:43:58.759966 5395 tablet_service.cc:1514] Processing DeleteTablet for tablet bc4a3b4dbddd442097cf75f68aa9c429 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:58 UTC) from {username='slave'} at 127.0.0.1:45380
I20260926 20:43:58.760195 5498 tablet_replica.cc:330] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898: stopping tablet replica
I20260926 20:43:58.760191 5133 tablet_service.cc:1514] Processing DeleteTablet for tablet bc4a3b4dbddd442097cf75f68aa9c429 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:58 UTC) from {username='slave'} at 127.0.0.1:42360
I20260926 20:43:58.760289 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 5071
I20260926 20:43:58.760372 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet bc4a3b4dbddd442097cf75f68aa9c429 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:43:58 UTC) from {username='slave'} at 127.0.0.1:58944
I20260926 20:43:58.760417 5498 raft_consensus.cc:2229] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:43:58.760619 5498 raft_consensus.cc:2258] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:58.760870 5500 tablet_replica.cc:330] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41: stopping tablet replica
I20260926 20:43:58.760962 5500 raft_consensus.cc:2229] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:43:58.761046 5500 raft_consensus.cc:2258] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:43:58.761399 5498 ts_tablet_manager.cc:1905] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:43:58.761490 5500 ts_tablet_manager.cc:1905] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:43:58.763226 5498 ts_tablet_manager.cc:1918] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:43:58.763360 5498 log.cc:1196] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/bc4a3b4dbddd442097cf75f68aa9c429
I20260926 20:43:58.763458 5500 ts_tablet_manager.cc:1918] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:43:58.763533 5500 log.cc:1196] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/bc4a3b4dbddd442097cf75f68aa9c429
I20260926 20:43:58.763831 5498 ts_tablet_manager.cc:1939] T bc4a3b4dbddd442097cf75f68aa9c429 P be9cb4898b1b449bb6f66c82b9186898: Deleting consensus metadata
I20260926 20:43:58.763851 5500 ts_tablet_manager.cc:1939] T bc4a3b4dbddd442097cf75f68aa9c429 P d474d5513d734e92998087f391f67e41: Deleting consensus metadata
I20260926 20:43:58.764246 4995 catalog_manager.cc:4841] TS d474d5513d734e92998087f391f67e41 (127.0.98.66:45361): tablet bc4a3b4dbddd442097cf75f68aa9c429 (table test-table [id=980458b1d2a44cf8bff0c25eb8db9f6b]) successfully deleted
I20260926 20:43:58.767063 4996 catalog_manager.cc:4841] TS be9cb4898b1b449bb6f66c82b9186898 (127.0.98.67:42271): tablet bc4a3b4dbddd442097cf75f68aa9c429 (table test-table [id=980458b1d2a44cf8bff0c25eb8db9f6b]) successfully deleted
W20260926 20:43:58.767256 4996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:40259 (error 108)
I20260926 20:43:58.767530 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 5202
I20260926 20:43:58.767700 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.767576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:40259 (local address 127.0.0.1:42376)
0926 20:43:58.767626 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:58.767662 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:40259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 20:43:58.768316 4996 catalog_manager.cc:4584] TS e62d95f3018d45c9be0c0acdb2dd09ba (127.0.98.65:40259): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc4a3b4dbddd442097cf75f68aa9c429: Network error: Client connection negotiation failed: client connection to 127.0.98.65:40259: connect: Connection refused (error 111)
I20260926 20:43:58.772257 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 5333
I20260926 20:43:58.776610 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 4980
2026-09-26T20:43:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (692 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 20:43:58.795691 393 mini_oidc.cc:152] Starting JWKS server
I20260926 20:43:58.797788 393 webserver.cc:460] Webserver started at https://127.0.0.1:46209/ using document root <none> and password file <none>
I20260926 20:43:58.797935 393 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:43:58.798120 393 webserver.cc:460] Webserver started at http://127.0.0.1:44565/ using document root <none> and password file <none>
2026-09-26T20:43:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:43:58Z Disabled control of system clock
I20260926 20:43:58.805848 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:42095
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:40755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:42095
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46209/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0/testchainca.cert with env {}
W20260926 20:43:58.887414 5507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:58.887645 5507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:58.887681 5507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:58.888986 5507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:43:58.889055 5507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:58.889077 5507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:58.889104 5507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:43:58.889137 5507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:43:58.889163 5507 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:43:58.889190 5507 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46209/jwks.json
I20260926 20:43:58.890673 5507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:40755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:42095
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:42095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46209/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:58.890942 5507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:58.891150 5507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:58.895665 5512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:58.895835 5515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:58.895975 5513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:58.896014 5507 server_base.cc:1034] running on GCE node
I20260926 20:43:58.896227 5507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:58.896456 5507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:58.897601 5507 hybrid_clock.cc:648] HybridClock initialized: now 1790455438897588 us; error 28 us; skew 500 ppm
I20260926 20:43:58.898813 5507 webserver.cc:460] Webserver started at http://127.0.98.126:44647/ using document root <none> and password file <none>
I20260926 20:43:58.899000 5507 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:58.899041 5507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:58.899127 5507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:58.900151 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "0ee53cb52f3b44c59b3730f1e541d60d"
format_stamp: "Formatted at 2026-09-26 20:43:58 on dist-test-slave-4hd3"
server_key: "4f502371b74c67d55063b212eae79930"
server_key_iv: "090c8cdd2b33e2af5401cf70e5c2b4c6"
server_key_version: "encryptionkey@0"
I20260926 20:43:58.900480 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "0ee53cb52f3b44c59b3730f1e541d60d"
format_stamp: "Formatted at 2026-09-26 20:43:58 on dist-test-slave-4hd3"
server_key: "4f502371b74c67d55063b212eae79930"
server_key_iv: "090c8cdd2b33e2af5401cf70e5c2b4c6"
server_key_version: "encryptionkey@0"
I20260926 20:43:58.901877 5507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:43:58.902839 5521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:58.903074 5507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:43:58.903162 5507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "0ee53cb52f3b44c59b3730f1e541d60d"
format_stamp: "Formatted at 2026-09-26 20:43:58 on dist-test-slave-4hd3"
server_key: "4f502371b74c67d55063b212eae79930"
server_key_iv: "090c8cdd2b33e2af5401cf70e5c2b4c6"
server_key_version: "encryptionkey@0"
I20260926 20:43:58.903229 5507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:58.927613 5507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:58.927969 5507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:58.928128 5507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:58.931978 5507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:42095
I20260926 20:43:58.932027 5573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:42095 every 8 connection(s)
I20260926 20:43:58.932399 5507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:43:58.933250 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 5507
I20260926 20:43:58.933354 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:43:58.933486 5574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:58.933566 393 external_mini_cluster.cc:1428] Setting key 657a095b9d664dff7a499838c0cdb31a
I20260926 20:43:58.936731 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d: Bootstrap starting.
I20260926 20:43:58.937423 5574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:58.937708 5574 log.cc:824] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:58.938567 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d: No bootstrap required, opened a new log
I20260926 20:43:58.939311 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.934114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43332 (local address 127.0.98.126:42095)
0926 20:43:58.934325 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:58.934332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:58.934359 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:43:58.934779 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:58.934785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:58.934843 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:58.934957 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:58.935186 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.936140 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.937270 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.937742 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.937878 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:58.938581 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:58.938584 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:58.938584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:43:58.938826 (+ 242us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:58.938863 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:43:58.938874 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:58.938926 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:58.939025 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 20:43:58.939108 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260926 20:43:58.940326 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:43:58.940970 5574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee53cb52f3b44c59b3730f1e541d60d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42095 } }
I20260926 20:43:58.941143 5574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:58.941187 5574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee53cb52f3b44c59b3730f1e541d60d, State: Initialized, Role: FOLLOWER
I20260926 20:43:58.941313 5574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee53cb52f3b44c59b3730f1e541d60d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42095 } }
I20260926 20:43:58.941392 5574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:43:58.941424 5574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:43:58.941469 5574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:58.941985 5574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee53cb52f3b44c59b3730f1e541d60d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42095 } }
I20260926 20:43:58.942103 5574 leader_election.cc:302] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ee53cb52f3b44c59b3730f1e541d60d; no voters:
I20260926 20:43:58.942278 5574 leader_election.cc:288] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:43:58.942315 5579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:58.942513 5579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d [term 1 LEADER]: Becoming Leader. State: Replica: 0ee53cb52f3b44c59b3730f1e541d60d, State: Running, Role: LEADER
I20260926 20:43:58.942569 5574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:43:58.942627 5579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee53cb52f3b44c59b3730f1e541d60d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42095 } }
I20260926 20:43:58.943012 5581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ee53cb52f3b44c59b3730f1e541d60d. Latest consensus state: current_term: 1 leader_uuid: "0ee53cb52f3b44c59b3730f1e541d60d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee53cb52f3b44c59b3730f1e541d60d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42095 } } }
I20260926 20:43:58.943115 5581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:58.943007 5580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ee53cb52f3b44c59b3730f1e541d60d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee53cb52f3b44c59b3730f1e541d60d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42095 } } }
I20260926 20:43:58.943289 5586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:43:58.943292 5580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:58.943835 5586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:43:58.945377 5586 catalog_manager.cc:1347] Generated new cluster ID: 5fb98b81405e4f489949d8db4e04c8db
I20260926 20:43:58.945461 5586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:43:58.964542 5586 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:43:58.965121 5586 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:43:58.971942 5586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ee53cb52f3b44c59b3730f1e541d60d: Generated new TSK 0
I20260926 20:43:58.972187 5586 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 20:43:59.001729 5604 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 20:43:59.001820 5604 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 20:43:59.002025 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:58.979093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43334 (local address 127.0.98.126:42095)
0926 20:43:58.979132 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:58.979135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:58.979271 (+ 136us) server_negotiation.cc:407] Connection header received
0926 20:43:58.979369 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:58.979369 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:58.979379 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:58.979441 (+ 62us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:43:58.979629 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.980170 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.981327 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.981507 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.981556 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:58.981787 (+ 231us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:43:59.001725 (+ 19938us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 20:43:59.001936 (+ 211us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43334: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 20:43:59.002080 5577 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43334: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 20:43:59.002360 5603 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:43:58.979032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42095 (local address 127.0.0.1:43334)
0926 20:43:58.979201 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:58.979213 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:58.979280 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:58.979483 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:58.979484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:58.979486 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:43:58.979565 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:58.979567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.980264 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.980266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:58.981232 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:58.981236 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:58.981579 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:58.981580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:58.981651 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:58.981674 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:43:59.001889 (+ 20215us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 20:43:59.002010 (+ 121us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.126:42095: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.126:42095: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 20:43:59.002893 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 5507
2026-09-26T20:43:59Z chronyd exiting
I20260926 20:43:59.038743 393 test_util.cc:175] -----------------------------------------------
I20260926 20:43:59.038874 393 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455416624047-393-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (244 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 20:43:59.039645 393 mini_oidc.cc:152] Starting JWKS server
I20260926 20:43:59.040382 393 webserver.cc:460] Webserver started at https://127.0.0.1:36361/ using document root <none> and password file <none>
I20260926 20:43:59.040455 393 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:43:59.040602 393 webserver.cc:460] Webserver started at http://127.0.0.1:46559/ using document root <none> and password file <none>
2026-09-26T20:43:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:43:59Z Disabled control of system clock
I20260926 20:43:59.048009 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:41935
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:43661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:41935
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36361/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455416624047-393-0/testchainca.cert with env {}
W20260926 20:43:59.126282 5610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:59.126480 5610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:59.126503 5610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:59.127666 5610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:43:59.127707 5610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:59.127744 5610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:59.127758 5610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:43:59.127799 5610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:43:59.127830 5610 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:43:59.127846 5610 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36361/jwks.json
I20260926 20:43:59.129367 5610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:41935
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:41935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36361/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455416624047-393-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:59.129630 5610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:59.129848 5610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:59.134579 5615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:59.134676 5610 server_base.cc:1034] running on GCE node
W20260926 20:43:59.134618 5618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:59.134594 5616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:59.135030 5610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:59.135325 5610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:59.136464 5610 hybrid_clock.cc:648] HybridClock initialized: now 1790455439136414 us; error 68 us; skew 500 ppm
I20260926 20:43:59.137737 5610 webserver.cc:460] Webserver started at http://127.0.98.126:36437/ using document root <none> and password file <none>
I20260926 20:43:59.138017 5610 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:59.138065 5610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:59.138228 5610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:59.139269 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "789b72ac2ea24cc79a8ddce7a434f71f"
format_stamp: "Formatted at 2026-09-26 20:43:59 on dist-test-slave-4hd3"
server_key: "6015493a8b11ffb34fc4c481d4495068"
server_key_iv: "63d09937e9423243a6f9843e71db2e61"
server_key_version: "encryptionkey@0"
I20260926 20:43:59.139627 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "789b72ac2ea24cc79a8ddce7a434f71f"
format_stamp: "Formatted at 2026-09-26 20:43:59 on dist-test-slave-4hd3"
server_key: "6015493a8b11ffb34fc4c481d4495068"
server_key_iv: "63d09937e9423243a6f9843e71db2e61"
server_key_version: "encryptionkey@0"
I20260926 20:43:59.141013 5610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:43:59.141762 5624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:59.141952 5610 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:43:59.142040 5610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "789b72ac2ea24cc79a8ddce7a434f71f"
format_stamp: "Formatted at 2026-09-26 20:43:59 on dist-test-slave-4hd3"
server_key: "6015493a8b11ffb34fc4c481d4495068"
server_key_iv: "63d09937e9423243a6f9843e71db2e61"
server_key_version: "encryptionkey@0"
I20260926 20:43:59.142114 5610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:59.157207 5610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:59.157516 5610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:59.157666 5610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:59.161518 5610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:41935
I20260926 20:43:59.161569 5676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:41935 every 8 connection(s)
I20260926 20:43:59.161918 5610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:43:59.162859 5677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:59.165112 5677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f: Bootstrap starting.
I20260926 20:43:59.165624 5677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:59.165850 5677 log.cc:824] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:59.166471 5677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f: No bootstrap required, opened a new log
I20260926 20:43:59.167979 5677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789b72ac2ea24cc79a8ddce7a434f71f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41935 } }
I20260926 20:43:59.168109 5677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:59.168131 5677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789b72ac2ea24cc79a8ddce7a434f71f, State: Initialized, Role: FOLLOWER
I20260926 20:43:59.168228 5677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789b72ac2ea24cc79a8ddce7a434f71f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41935 } }
I20260926 20:43:59.168310 5677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:43:59.168344 5677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:43:59.168404 5677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:59.168928 5677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789b72ac2ea24cc79a8ddce7a434f71f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41935 } }
I20260926 20:43:59.169044 5677 leader_election.cc:302] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 789b72ac2ea24cc79a8ddce7a434f71f; no voters:
I20260926 20:43:59.169222 5677 leader_election.cc:288] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:43:59.169265 5680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:59.169454 5680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f [term 1 LEADER]: Becoming Leader. State: Replica: 789b72ac2ea24cc79a8ddce7a434f71f, State: Running, Role: LEADER
I20260926 20:43:59.169502 5677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:43:59.169554 5680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789b72ac2ea24cc79a8ddce7a434f71f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41935 } }
I20260926 20:43:59.169938 5681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "789b72ac2ea24cc79a8ddce7a434f71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789b72ac2ea24cc79a8ddce7a434f71f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41935 } } }
I20260926 20:43:59.170047 5682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 789b72ac2ea24cc79a8ddce7a434f71f. Latest consensus state: current_term: 1 leader_uuid: "789b72ac2ea24cc79a8ddce7a434f71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789b72ac2ea24cc79a8ddce7a434f71f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41935 } } }
I20260926 20:43:59.170121 5682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:59.170037 5681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:59.170378 5689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:43:59.170936 5689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:43:59.171907 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 5610
I20260926 20:43:59.172000 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:43:59.172227 393 external_mini_cluster.cc:1428] Setting key 4a3f6310a13bd59965eeeeabfe637a42
I20260926 20:43:59.172544 5689 catalog_manager.cc:1347] Generated new cluster ID: 3b1b6df4072f466ca75f2800f9289d21
I20260926 20:43:59.172592 5689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:43:59.181928 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:59.172828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56538 (local address 127.0.98.126:41935)
0926 20:43:59.173035 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:59.173043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:59.176775 (+ 3732us) server_negotiation.cc:407] Connection header received
0926 20:43:59.176938 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:59.176944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:59.176995 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:59.177129 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:59.177322 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:59.179153 (+ 1831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:59.180030 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:59.180432 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:59.180571 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:59.181256 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:59.181260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:59.181260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:43:59.181441 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:59.181472 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:43:59.181482 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:59.181552 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:59.181632 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 20:43:59.181715 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":101,"threads_started":1}
I20260926 20:43:59.182860 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:43:59.186970 5689 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:43:59.187620 5689 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:43:59.197743 5689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 789b72ac2ea24cc79a8ddce7a434f71f: Generated new TSK 0
I20260926 20:43:59.197954 5689 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 20:43:59.216202 5707 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 20:43:59.216276 5707 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 20:43:59.216641 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:59.202690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.0.98.126:41935)
0926 20:43:59.202726 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:59.202728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:59.202846 (+ 118us) server_negotiation.cc:407] Connection header received
0926 20:43:59.202933 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:59.202934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:59.202945 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:59.203037 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:43:59.203171 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:59.203676 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:59.204495 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:59.204706 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:59.204792 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:59.204981 (+ 189us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:43:59.216412 (+ 11431us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36361/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 20:43:59.216565 (+ 153us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56554: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36361/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 20:43:59.216701 5706 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:43:59.202621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41935 (local address 127.0.0.1:56554)
0926 20:43:59.202780 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:59.202789 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:59.202863 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:59.203037 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:59.203038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:59.203039 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:43:59.203096 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:59.203098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:59.203804 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:59.203805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:59.204399 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:59.204401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:59.204795 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:59.204796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:59.204851 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:59.204856 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:43:59.216513 (+ 11657us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36361/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 20:43:59.216604 (+ 91us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.126:41935: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36361/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
W20260926 20:43:59.216698 5700 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56554: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36361/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260926 20:43:59.217023 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 5610
2026-09-26T20:43:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (214 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 20:43:59.254319 393 mini_oidc.cc:152] Starting JWKS server
I20260926 20:43:59.254968 393 webserver.cc:460] Webserver started at https://127.0.0.1:33485/ using document root <none> and password file <none>
I20260926 20:43:59.255008 393 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:43:59.255143 393 webserver.cc:460] Webserver started at http://127.0.0.1:41703/ using document root <none> and password file <none>
2026-09-26T20:43:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:43:59Z Disabled control of system clock
I20260926 20:43:59.262920 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:32831
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:32831
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33485/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 20:43:59.341317 5713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:59.341522 5713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:59.341543 5713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:59.342654 5713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:43:59.342691 5713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:59.342705 5713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:59.342716 5713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:43:59.342728 5713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:43:59.342741 5713 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:43:59.342753 5713 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33485/jwks.json
I20260926 20:43:59.344066 5713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:32831
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:32831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33485/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:59.344262 5713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:59.344492 5713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:59.349232 5719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:59.349207 5718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:59.349206 5721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:59.349494 5713 server_base.cc:1034] running on GCE node
I20260926 20:43:59.349742 5713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:59.350041 5713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:59.351243 5713 hybrid_clock.cc:648] HybridClock initialized: now 1790455439351221 us; error 51 us; skew 500 ppm
I20260926 20:43:59.352667 5713 webserver.cc:460] Webserver started at http://127.0.98.126:44333/ using document root <none> and password file <none>
I20260926 20:43:59.352911 5713 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:59.352982 5713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:59.353135 5713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:59.354197 5713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "b2935b0d298b4de4a8d59dca23411194"
format_stamp: "Formatted at 2026-09-26 20:43:59 on dist-test-slave-4hd3"
server_key: "afe99c0eab0b89a444e00884d5ba77a0"
server_key_iv: "f18fbf3dcafbb2c76174bf28984cc9be"
server_key_version: "encryptionkey@0"
I20260926 20:43:59.354612 5713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "b2935b0d298b4de4a8d59dca23411194"
format_stamp: "Formatted at 2026-09-26 20:43:59 on dist-test-slave-4hd3"
server_key: "afe99c0eab0b89a444e00884d5ba77a0"
server_key_iv: "f18fbf3dcafbb2c76174bf28984cc9be"
server_key_version: "encryptionkey@0"
I20260926 20:43:59.356024 5713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:43:59.356786 5727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:59.356961 5713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:43:59.357060 5713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "b2935b0d298b4de4a8d59dca23411194"
format_stamp: "Formatted at 2026-09-26 20:43:59 on dist-test-slave-4hd3"
server_key: "afe99c0eab0b89a444e00884d5ba77a0"
server_key_iv: "f18fbf3dcafbb2c76174bf28984cc9be"
server_key_version: "encryptionkey@0"
I20260926 20:43:59.357143 5713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:59.377171 5713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:59.377508 5713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:59.377678 5713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:59.381841 5713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:32831
I20260926 20:43:59.381879 5779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:32831 every 8 connection(s)
I20260926 20:43:59.382246 5713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:43:59.382899 5780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:59.385556 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194: Bootstrap starting.
I20260926 20:43:59.386101 5780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:59.386335 5780 log.cc:824] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:59.386991 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194: No bootstrap required, opened a new log
I20260926 20:43:59.387017 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 5713
I20260926 20:43:59.387125 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:43:59.387295 393 external_mini_cluster.cc:1428] Setting key 85c3b6248121a38e6eca22aeff905d8a
I20260926 20:43:59.388943 5780 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2935b0d298b4de4a8d59dca23411194" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32831 } }
I20260926 20:43:59.389119 5780 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:59.389168 5780 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2935b0d298b4de4a8d59dca23411194, State: Initialized, Role: FOLLOWER
I20260926 20:43:59.389292 5780 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2935b0d298b4de4a8d59dca23411194" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32831 } }
I20260926 20:43:59.389392 5780 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:43:59.389438 5780 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:43:59.389493 5780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:59.390044 5780 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2935b0d298b4de4a8d59dca23411194" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32831 } }
I20260926 20:43:59.390149 5780 leader_election.cc:302] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2935b0d298b4de4a8d59dca23411194; no voters:
I20260926 20:43:59.390300 5780 leader_election.cc:288] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:43:59.390331 5785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:59.390439 5785 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194 [term 1 LEADER]: Becoming Leader. State: Replica: b2935b0d298b4de4a8d59dca23411194, State: Running, Role: LEADER
I20260926 20:43:59.390525 5785 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2935b0d298b4de4a8d59dca23411194" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32831 } }
I20260926 20:43:59.390554 5780 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:43:59.391059 5786 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2935b0d298b4de4a8d59dca23411194. Latest consensus state: current_term: 1 leader_uuid: "b2935b0d298b4de4a8d59dca23411194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2935b0d298b4de4a8d59dca23411194" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32831 } } }
I20260926 20:43:59.391072 5787 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2935b0d298b4de4a8d59dca23411194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2935b0d298b4de4a8d59dca23411194" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32831 } } }
I20260926 20:43:59.391227 5786 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194 [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:59.391244 5787 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194 [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:59.391572 5794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:43:59.392091 5794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:43:59.394136 5794 catalog_manager.cc:1347] Generated new cluster ID: c87cf955a36c40498791c8670a1f4eda
I20260926 20:43:59.394193 5794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:43:59.394637 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:59.387858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34818 (local address 127.0.98.126:32831)
0926 20:43:59.388069 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:59.388077 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:59.388102 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:43:59.388940 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:59.388948 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:59.389012 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:59.389182 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:59.389357 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:59.391022 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:59.392199 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:59.392872 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:59.393029 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:59.393764 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:59.393767 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:59.393768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:43:59.393973 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:59.394012 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:43:59.394028 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:59.394088 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:59.394359 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 20:43:59.394433 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260926 20:43:59.395573 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:43:59.401265 5794 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:43:59.401748 5794 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:43:59.404430 5794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2935b0d298b4de4a8d59dca23411194: Generated new TSK 0
I20260926 20:43:59.404595 5794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:43:59.416658 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:59.413155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34826 (local address 127.0.98.126:32831)
0926 20:43:59.413217 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:59.413219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:59.413331 (+ 112us) server_negotiation.cc:407] Connection header received
0926 20:43:59.413397 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:59.413397 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:59.413408 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:59.413477 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:43:59.413598 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:59.414039 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:59.415178 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:59.415351 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:59.415402 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:59.415585 (+ 183us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:43:59.416418 (+ 833us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33485/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 20:43:59.416580 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34826: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33485/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 20:43:59.416723 5809 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:43:59.413092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32831 (local address 127.0.0.1:34826)
0926 20:43:59.413249 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:59.413260 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:59.413332 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:59.413481 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:59.413482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:59.413484 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:43:59.413537 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:59.413538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:59.414131 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:59.414132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:59.415094 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:59.415097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:59.415428 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:59.415430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:59.415488 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:59.415493 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:43:59.416538 (+ 1045us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33485/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 20:43:59.416624 (+ 86us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.126:32831: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33485/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":108,"threads_started":1}
W20260926 20:43:59.416723 5784 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34826: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33485/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260926 20:43:59.416980 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 5713
E20260926 20:43:59.427505 5810 webserver.cc:533] Webserver: sslize failed:
E20260926 20:43:59.427570 5810 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T20:43:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (199 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T20:43:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:43:59Z Disabled control of system clock
I20260926 20:43:59.461645 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37305
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:36727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:37305
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455416624047-393-0/testchainca.cert with env {}
W20260926 20:43:59.542598 5814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:43:59.542805 5814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:43:59.542827 5814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:43:59.544067 5814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:43:59.544113 5814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:43:59.544130 5814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:43:59.544164 5814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:43:59.544189 5814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:43:59.544211 5814 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:43:59.544237 5814 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 20:43:59.545714 5814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:37305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455416624047-393-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:43:59.545974 5814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:43:59.546193 5814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:43:59.550945 5820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:59.551123 5822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:43:59.551126 5819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:43:59.551322 5814 server_base.cc:1034] running on GCE node
I20260926 20:43:59.551610 5814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:43:59.551952 5814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:43:59.553155 5814 hybrid_clock.cc:648] HybridClock initialized: now 1790455439553141 us; error 45 us; skew 500 ppm
I20260926 20:43:59.554567 5814 webserver.cc:460] Webserver started at http://127.0.98.126:46695/ using document root <none> and password file <none>
I20260926 20:43:59.554826 5814 fs_manager.cc:359] Metadata directory not provided
I20260926 20:43:59.554899 5814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:43:59.555045 5814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:43:59.556185 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "1ce9e7e100534149b6051bff3a2e8023"
format_stamp: "Formatted at 2026-09-26 20:43:59 on dist-test-slave-4hd3"
server_key: "4fb69c24cb65e18db451b56ff97e5d44"
server_key_iv: "3b56517573495855df38c3c70862ca29"
server_key_version: "encryptionkey@0"
I20260926 20:43:59.556537 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "1ce9e7e100534149b6051bff3a2e8023"
format_stamp: "Formatted at 2026-09-26 20:43:59 on dist-test-slave-4hd3"
server_key: "4fb69c24cb65e18db451b56ff97e5d44"
server_key_iv: "3b56517573495855df38c3c70862ca29"
server_key_version: "encryptionkey@0"
I20260926 20:43:59.557835 5814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:43:59.558516 5828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:43:59.558686 5814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:43:59.558794 5814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "1ce9e7e100534149b6051bff3a2e8023"
format_stamp: "Formatted at 2026-09-26 20:43:59 on dist-test-slave-4hd3"
server_key: "4fb69c24cb65e18db451b56ff97e5d44"
server_key_iv: "3b56517573495855df38c3c70862ca29"
server_key_version: "encryptionkey@0"
I20260926 20:43:59.558866 5814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:43:59.580847 5814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:43:59.581162 5814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:43:59.581321 5814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:43:59.585172 5814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:37305
I20260926 20:43:59.585208 5880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:37305 every 8 connection(s)
I20260926 20:43:59.585547 5814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:43:59.586095 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 5814
I20260926 20:43:59.586200 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:43:59.586404 393 external_mini_cluster.cc:1428] Setting key 659cb60ee14fcba79e7b9f45d354776e
I20260926 20:43:59.586752 5881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:43:59.589532 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023: Bootstrap starting.
I20260926 20:43:59.590129 5881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023: Neither blocks nor log segments found. Creating new log.
I20260926 20:43:59.590426 5881 log.cc:824] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023: Log is configured to *not* fsync() on all Append() calls
I20260926 20:43:59.591271 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023: No bootstrap required, opened a new log
I20260926 20:43:59.592095 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:59.586914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44384 (local address 127.0.98.126:37305)
0926 20:43:59.587155 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:59.587162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:59.587187 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:43:59.587885 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:59.587892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:59.587957 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:59.588106 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:43:59.588344 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:59.589197 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:59.590047 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:59.590495 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:59.590630 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:59.591389 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:43:59.591393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:43:59.591393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:43:59.591552 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:43:59.591589 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:43:59.591600 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:43:59.591651 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:43:59.591819 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 20:43:59.591923 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":102,"threads_started":1}
I20260926 20:43:59.593071 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:43:59.593628 5881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce9e7e100534149b6051bff3a2e8023" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37305 } }
I20260926 20:43:59.593803 5881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:43:59.593839 5881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ce9e7e100534149b6051bff3a2e8023, State: Initialized, Role: FOLLOWER
I20260926 20:43:59.593952 5881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce9e7e100534149b6051bff3a2e8023" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37305 } }
I20260926 20:43:59.594035 5881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:43:59.594065 5881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:43:59.594110 5881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:43:59.594632 5881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce9e7e100534149b6051bff3a2e8023" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37305 } }
I20260926 20:43:59.594713 5881 leader_election.cc:302] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ce9e7e100534149b6051bff3a2e8023; no voters:
I20260926 20:43:59.594859 5881 leader_election.cc:288] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:43:59.594897 5886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:43:59.595016 5886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023 [term 1 LEADER]: Becoming Leader. State: Replica: 1ce9e7e100534149b6051bff3a2e8023, State: Running, Role: LEADER
I20260926 20:43:59.595116 5881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:43:59.595094 5886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce9e7e100534149b6051bff3a2e8023" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37305 } }
I20260926 20:43:59.595456 5888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ce9e7e100534149b6051bff3a2e8023. Latest consensus state: current_term: 1 leader_uuid: "1ce9e7e100534149b6051bff3a2e8023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce9e7e100534149b6051bff3a2e8023" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37305 } } }
I20260926 20:43:59.595551 5888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023 [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:59.595743 5887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ce9e7e100534149b6051bff3a2e8023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce9e7e100534149b6051bff3a2e8023" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37305 } } }
I20260926 20:43:59.595816 5887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023 [sys.catalog]: This master's current role is: LEADER
I20260926 20:43:59.596133 5892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:43:59.596748 5892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:43:59.598301 5892 catalog_manager.cc:1347] Generated new cluster ID: 63a2aca39cec4782be0cbfbb5cf766a3
I20260926 20:43:59.598364 5892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:43:59.610831 5892 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:43:59.611531 5892 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:43:59.614676 5892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ce9e7e100534149b6051bff3a2e8023: Generated new TSK 0
I20260926 20:43:59.614837 5892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:44:00.057528 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:43:59.624014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44392 (local address 127.0.98.126:37305)
0926 20:43:59.624067 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:43:59.624069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:43:59.624173 (+ 104us) server_negotiation.cc:407] Connection header received
0926 20:43:59.624289 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:43:59.624290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:43:59.624303 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:43:59.624389 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:43:59.624545 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:59.625344 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:59.626105 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:43:59.626422 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:43:59.626492 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:59.626643 (+ 151us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:44:00.057267 (+430624us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 20:44:00.057460 (+ 193us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44392: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 20:44:00.057593 5884 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44392: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 20:44:00.057708 5910 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:43:59.623923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37305 (local address 127.0.0.1:44392)
0926 20:43:59.624103 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 20:43:59.624115 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:43:59.624180 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:43:59.624407 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:43:59.624408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:43:59.624411 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:43:59.624486 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:59.624488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:59.625430 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:59.625431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:59.626030 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:43:59.626032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:43:59.626509 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:43:59.626510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:43:59.626548 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:43:59.626553 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:44:00.057426 (+430873us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 20:44:00.057548 (+ 122us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.126:37305: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":93,"threads_started":1}
I20260926 20:44:00.058111 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 5814
2026-09-26T20:44:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (621 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (93 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:44:00 dist-test-slave-4hd3 krb5kdc[5922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:44:00 dist-test-slave-4hd3 krb5kdc[5922](info): set up 2 sockets
Sep 26 20:44:00 dist-test-slave-4hd3 krb5kdc[5922](info): commencing operation
krb5kdc: starting...
W20260926 20:44:01.271970 393 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:44:01 dist-test-slave-4hd3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:44:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:44:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:44:01.390179 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38091
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:38091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:01.468222 5935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:01.468442 5935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:01.468466 5935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:01.469574 5935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:44:01.469607 5935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:01.469621 5935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:01.469635 5935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:44:01.469646 5935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:44:01.471076 5935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:38091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:01.471263 5935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:01.471436 5935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:01.476159 5943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:01.476203 5940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:01.476164 5941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:01.476506 5935 server_base.cc:1034] running on GCE node
I20260926 20:44:01.476783 5935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:01.477087 5935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:01.478262 5935 hybrid_clock.cc:648] HybridClock initialized: now 1790455441478247 us; error 27 us; skew 500 ppm
Sep 26 20:44:01 dist-test-slave-4hd3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:01.485366 5935 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:44:01.485860 5935 webserver.cc:460] Webserver started at http://127.0.98.126:40943/ using document root <none> and password file <none>
I20260926 20:44:01.486079 5935 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:01.486147 5935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:01.486291 5935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:01.487397 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "e16beef445ba49e980512b34aa498a4f"
format_stamp: "Formatted at 2026-09-26 20:44:01 on dist-test-slave-4hd3"
server_key: "939a732ba05e29c1b56b0c6b106650f8"
server_key_iv: "a58333863bbe943999ba12fe1a8fdecb"
server_key_version: "encryptionkey@0"
I20260926 20:44:01.487811 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "e16beef445ba49e980512b34aa498a4f"
format_stamp: "Formatted at 2026-09-26 20:44:01 on dist-test-slave-4hd3"
server_key: "939a732ba05e29c1b56b0c6b106650f8"
server_key_iv: "a58333863bbe943999ba12fe1a8fdecb"
server_key_version: "encryptionkey@0"
I20260926 20:44:01.489113 5935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:44:01.489867 5950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:01.490038 5935 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:44:01.490132 5935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "e16beef445ba49e980512b34aa498a4f"
format_stamp: "Formatted at 2026-09-26 20:44:01 on dist-test-slave-4hd3"
server_key: "939a732ba05e29c1b56b0c6b106650f8"
server_key_iv: "a58333863bbe943999ba12fe1a8fdecb"
server_key_version: "encryptionkey@0"
I20260926 20:44:01.490217 5935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:01.509018 5935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:01.511341 5935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:01.511507 5935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:01.515483 5935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:38091
I20260926 20:44:01.515520 6002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:38091 every 8 connection(s)
I20260926 20:44:01.516050 5935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:44:01.516713 6003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:01.519017 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f: Bootstrap starting.
I20260926 20:44:01.519606 6003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:01.519948 6003 log.cc:824] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:44:01.520604 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f: No bootstrap required, opened a new log
I20260926 20:44:01.522141 6003 raft_consensus.cc:348] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16beef445ba49e980512b34aa498a4f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38091 } }
I20260926 20:44:01.522298 6003 raft_consensus.cc:374] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:01.522347 6003 raft_consensus.cc:729] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16beef445ba49e980512b34aa498a4f, State: Initialized, Role: FOLLOWER
I20260926 20:44:01.522502 6003 consensus_queue.cc:260] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16beef445ba49e980512b34aa498a4f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38091 } }
I20260926 20:44:01.522573 6003 raft_consensus.cc:388] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:44:01.522614 6003 raft_consensus.cc:482] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:44:01.522662 6003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:01.523538 6003 raft_consensus.cc:504] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16beef445ba49e980512b34aa498a4f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38091 } }
I20260926 20:44:01.523653 6003 leader_election.cc:302] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e16beef445ba49e980512b34aa498a4f; no voters:
I20260926 20:44:01.523847 6003 leader_election.cc:288] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:44:01.523931 6006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:44:01.524099 6006 raft_consensus.cc:686] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f [term 1 LEADER]: Becoming Leader. State: Replica: e16beef445ba49e980512b34aa498a4f, State: Running, Role: LEADER
I20260926 20:44:01.524115 6003 sys_catalog.cc:564] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:44:01.524214 6006 consensus_queue.cc:237] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16beef445ba49e980512b34aa498a4f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38091 } }
I20260926 20:44:01.524593 6008 sys_catalog.cc:455] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e16beef445ba49e980512b34aa498a4f. Latest consensus state: current_term: 1 leader_uuid: "e16beef445ba49e980512b34aa498a4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16beef445ba49e980512b34aa498a4f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38091 } } }
I20260926 20:44:01.524676 6008 sys_catalog.cc:458] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:01.524797 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 5935
I20260926 20:44:01.524909 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:44:01.525115 393 external_mini_cluster.cc:1428] Setting key b9b059018a7403eb9f4126413a4c7ad2
I20260926 20:44:01.524911 6007 sys_catalog.cc:455] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e16beef445ba49e980512b34aa498a4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16beef445ba49e980512b34aa498a4f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38091 } } }
I20260926 20:44:01.525382 6007 sys_catalog.cc:458] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:01.525563 6015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:44:01.526067 6015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:44:01.527673 6015 catalog_manager.cc:1347] Generated new cluster ID: 98590a0fdcdf413bac5653844eb44d11
I20260926 20:44:01.527750 6015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:44:01 dist-test-slave-4hd3 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:44:01.536924 6025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:01.525788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.0.98.126:38091)
0926 20:44:01.525960 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:01.525970 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:01.526077 (+ 107us) server_negotiation.cc:407] Connection header received
0926 20:44:01.526275 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:01.526282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:01.526344 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:01.526486 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:01.527310 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:01.528666 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:01.529773 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:01.530370 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:01.530466 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:01.533265 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:01.533269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:01.533269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:01.533288 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:01.535259 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:01.535648 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:01.535653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:01.535653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:01.535811 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:01.536024 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:01.536025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:01.536025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:01.536344 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:01.536496 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:01.536572 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 20:44:01.536711 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260926 20:44:01.543179 6015 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:44:01.543699 6015 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:44:01.546701 6015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e16beef445ba49e980512b34aa498a4f: Generated new TSK 0
I20260926 20:44:01.546900 6015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260926 20:44:01.572247 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38091
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:01.654517 6029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:01.654717 6029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:01.654742 6029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:01.655907 6029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:01.655947 6029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:01.655974 6029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:44:01.657511 6029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:01.657703 6029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:01.657889 6029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:01.662791 6034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:01.662817 6037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:01.662917 6035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:01.663084 6029 server_base.cc:1034] running on GCE node
I20260926 20:44:01.663257 6029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:01.663499 6029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:01.664660 6029 hybrid_clock.cc:648] HybridClock initialized: now 1790455441664627 us; error 55 us; skew 500 ppm
Sep 26 20:44:01 dist-test-slave-4hd3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:01.667171 6029 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:44:01.667663 6029 webserver.cc:460] Webserver started at http://127.0.98.65:44177/ using document root <none> and password file <none>
I20260926 20:44:01.667929 6029 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:01.668001 6029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:01.668147 6029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:01.669163 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "59c36950c1354bf0be19504875e44878"
format_stamp: "Formatted at 2026-09-26 20:44:01 on dist-test-slave-4hd3"
server_key: "aef89d8989a7ad8f31fcf11d18394d46"
server_key_iv: "53f5b06d829ed5aba89bbcfc3a1de6da"
server_key_version: "encryptionkey@0"
I20260926 20:44:01.669538 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "59c36950c1354bf0be19504875e44878"
format_stamp: "Formatted at 2026-09-26 20:44:01 on dist-test-slave-4hd3"
server_key: "aef89d8989a7ad8f31fcf11d18394d46"
server_key_iv: "53f5b06d829ed5aba89bbcfc3a1de6da"
server_key_version: "encryptionkey@0"
I20260926 20:44:01.670833 6029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:44:01.671648 6044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:01.671823 6029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:01.671947 6029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "59c36950c1354bf0be19504875e44878"
format_stamp: "Formatted at 2026-09-26 20:44:01 on dist-test-slave-4hd3"
server_key: "aef89d8989a7ad8f31fcf11d18394d46"
server_key_iv: "53f5b06d829ed5aba89bbcfc3a1de6da"
server_key_version: "encryptionkey@0"
I20260926 20:44:01.672022 6029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:01.681346 6029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:01.683166 6029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:01.683331 6029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:01.683548 6029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:01.683911 6029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:01.683966 6029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:01.684006 6029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:01.684033 6029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:01.689895 6029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:38575
I20260926 20:44:01.689932 6157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:38575 every 8 connection(s)
I20260926 20:44:01.690235 6029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
Sep 26 20:44:01 dist-test-slave-4hd3 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:44:01.697289 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 6029
I20260926 20:44:01.697391 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:44:01.697592 393 external_mini_cluster.cc:1428] Setting key 84d2b7a3a38d87a51bd6db373213676c
I20260926 20:44:01.700213 6025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:01.690894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:33903 (local address 127.0.98.126:38091)
0926 20:44:01.690936 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:01.690938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:01.691124 (+ 186us) server_negotiation.cc:407] Connection header received
0926 20:44:01.692178 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:01.692179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:01.692194 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:01.692322 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:01.692956 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:01.693503 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:01.694489 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:01.694684 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:01.694746 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:01.696976 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:01.696977 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:01.696978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:01.696985 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:01.698671 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:01.699157 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:01.699158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:01.699158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:01.699207 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:01.699488 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:01.699488 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:01.699488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:01.699614 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:01.699690 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:01.700015 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 20:44:01.700119 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:44:01.700397 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:01.690879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38091 (local address 127.0.98.65:33903)
0926 20:44:01.691004 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:01.691038 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:01.692041 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:01.692370 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:01.692373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:01.692614 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:01.692795 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:01.692799 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:01.693592 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:01.693592 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:01.694408 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:01.694410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:01.694759 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:01.694759 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:01.694890 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:01.695332 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:01.695340 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:01.696851 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:01.698754 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:01.698760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:01.698762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:01.699083 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:01.699273 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:01.699274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:01.699274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:01.699425 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:01.699697 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:01.699704 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:01.699877 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 20:44:01.699994 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 20:44:01.700148 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":68,"threads_started":1}
I20260926 20:44:01.700917 6158 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38091
I20260926 20:44:01.701025 6158 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:01.701192 6158 heartbeater.cc:511] Master 127.0.98.126:38091 requested a full tablet report, sending...
I20260926 20:44:01.701731 5967 ts_manager.cc:194] Registered new tserver with Master: 59c36950c1354bf0be19504875e44878 (127.0.98.65:38575)
I20260926 20:44:01.702689 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:33903
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260926 20:44:01.723078 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38091
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:01.804185 6163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:01.804380 6163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:01.804402 6163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:01.805541 6163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:01.805584 6163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:01.805637 6163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:44:01.807082 6163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:01.807322 6163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:01.807565 6163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:01.812462 6171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:01.812563 6168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:01.812462 6169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:01.812963 6163 server_base.cc:1034] running on GCE node
I20260926 20:44:01.813261 6163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:01.813541 6163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:01.814697 6163 hybrid_clock.cc:648] HybridClock initialized: now 1790455441814678 us; error 39 us; skew 500 ppm
Sep 26 20:44:01 dist-test-slave-4hd3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:01.817570 6163 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:44:01.818097 6163 webserver.cc:460] Webserver started at http://127.0.98.66:33601/ using document root <none> and password file <none>
I20260926 20:44:01.818357 6163 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:01.818406 6163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:01.818589 6163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:01.819802 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "ef5bdcd8f15346af862c62402937019b"
format_stamp: "Formatted at 2026-09-26 20:44:01 on dist-test-slave-4hd3"
server_key: "383b2bc3db7c0ae35518e7c3f624961d"
server_key_iv: "861233b060e552e484aa8ebb9549aa90"
server_key_version: "encryptionkey@0"
I20260926 20:44:01.820188 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "ef5bdcd8f15346af862c62402937019b"
format_stamp: "Formatted at 2026-09-26 20:44:01 on dist-test-slave-4hd3"
server_key: "383b2bc3db7c0ae35518e7c3f624961d"
server_key_iv: "861233b060e552e484aa8ebb9549aa90"
server_key_version: "encryptionkey@0"
I20260926 20:44:01.821579 6163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:44:01.822376 6178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:01.822553 6163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:44:01.822649 6163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "ef5bdcd8f15346af862c62402937019b"
format_stamp: "Formatted at 2026-09-26 20:44:01 on dist-test-slave-4hd3"
server_key: "383b2bc3db7c0ae35518e7c3f624961d"
server_key_iv: "861233b060e552e484aa8ebb9549aa90"
server_key_version: "encryptionkey@0"
I20260926 20:44:01.822729 6163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:01.870062 6163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:01.872155 6163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:01.872329 6163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:01.872586 6163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:01.872936 6163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:01.872972 6163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:01.872996 6163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:01.873046 6163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:01.879465 6163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:33271
I20260926 20:44:01.879515 6291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:33271 every 8 connection(s)
I20260926 20:44:01.879887 6163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:44:01 dist-test-slave-4hd3 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:44:01.888741 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 6163
I20260926 20:44:01.888844 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:44:01.889055 393 external_mini_cluster.cc:1428] Setting key 121101e9f15620c97f32cde9dc0ebc37
I20260926 20:44:01.889309 6025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:01.880581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:43855 (local address 127.0.98.126:38091)
0926 20:44:01.880659 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:01.880661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:01.880859 (+ 198us) server_negotiation.cc:407] Connection header received
0926 20:44:01.881934 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:01.881935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:01.881946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:01.882018 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:01.882613 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:01.883096 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:01.884217 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:01.884427 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:01.884496 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:01.886605 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:01.886606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:01.886606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:01.886613 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:01.887896 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:01.888285 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:01.888286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:01.888286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:01.888335 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:01.888598 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:01.888599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:01.888600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:01.888759 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:01.888838 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:01.889154 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 20:44:01.889246 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:44:01.889576 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:01.880552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38091 (local address 127.0.98.66:43855)
0926 20:44:01.880719 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:01.880761 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:01.881751 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:01.882078 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:01.882081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:01.882341 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:01.882537 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:01.882542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:01.883171 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:01.883171 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:01.884132 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:01.884134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:01.884517 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:01.884518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:01.884616 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:01.885086 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:01.885093 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:01.886516 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:01.887986 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:01.887992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:01.887994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:01.888210 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:01.888414 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:01.888415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:01.888415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:01.888526 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:01.888871 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:01.888877 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:01.889013 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 20:44:01.889132 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 20:44:01.889311 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260926 20:44:01.890123 6292 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38091
I20260926 20:44:01.890219 6292 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:01.890407 6292 heartbeater.cc:511] Master 127.0.98.126:38091 requested a full tablet report, sending...
I20260926 20:44:01.890781 5967 ts_manager.cc:194] Registered new tserver with Master: ef5bdcd8f15346af862c62402937019b (127.0.98.66:33271)
I20260926 20:44:01.891335 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:43855
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260926 20:44:01.914673 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38091
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:01.994481 6297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:01.994679 6297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:01.994702 6297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:01.995820 6297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:01.995855 6297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:01.995880 6297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:44:01.997356 6297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:01.997543 6297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:01.997709 6297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:02.002496 6302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:02.002472 6303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:02.002593 6305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:02.002780 6297 server_base.cc:1034] running on GCE node
I20260926 20:44:02.003031 6297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:02.003291 6297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:02.004479 6297 hybrid_clock.cc:648] HybridClock initialized: now 1790455442004462 us; error 26 us; skew 500 ppm
Sep 26 20:44:02 dist-test-slave-4hd3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:02.007061 6297 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:44:02.007530 6297 webserver.cc:460] Webserver started at http://127.0.98.67:40993/ using document root <none> and password file <none>
I20260926 20:44:02.007762 6297 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:02.007822 6297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:02.007912 6297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:02.008884 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "9cde82a13f464feaa8401a762593acbb"
format_stamp: "Formatted at 2026-09-26 20:44:02 on dist-test-slave-4hd3"
server_key: "785926b6c4ecd3850e704e5c442b3773"
server_key_iv: "887f899f27e2c57b4eee320d6d3ce0f4"
server_key_version: "encryptionkey@0"
I20260926 20:44:02.009266 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "9cde82a13f464feaa8401a762593acbb"
format_stamp: "Formatted at 2026-09-26 20:44:02 on dist-test-slave-4hd3"
server_key: "785926b6c4ecd3850e704e5c442b3773"
server_key_iv: "887f899f27e2c57b4eee320d6d3ce0f4"
server_key_version: "encryptionkey@0"
I20260926 20:44:02.010541 6297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:44:02.011279 6312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:02.011480 6297 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:02.011584 6297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "9cde82a13f464feaa8401a762593acbb"
format_stamp: "Formatted at 2026-09-26 20:44:02 on dist-test-slave-4hd3"
server_key: "785926b6c4ecd3850e704e5c442b3773"
server_key_iv: "887f899f27e2c57b4eee320d6d3ce0f4"
server_key_version: "encryptionkey@0"
I20260926 20:44:02.011659 6297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:02.019125 6297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:02.021145 6297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:02.021310 6297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:02.021534 6297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:02.021836 6297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:02.021890 6297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:02.021926 6297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:02.021955 6297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:02.028187 6297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:39845
I20260926 20:44:02.028221 6425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:39845 every 8 connection(s)
I20260926 20:44:02.028542 6297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:44:02.029215 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 6297
I20260926 20:44:02.029320 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:44:02.029534 393 external_mini_cluster.cc:1428] Setting key 52730c9ceec6f9af245a64766e011d59
Sep 26 20:44:02 dist-test-slave-4hd3 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:44:02.038107 6025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:02.029371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:41897 (local address 127.0.98.126:38091)
0926 20:44:02.029440 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:02.029442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:02.029726 (+ 284us) server_negotiation.cc:407] Connection header received
0926 20:44:02.030909 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:02.030910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:02.030920 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:02.030967 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:02.031527 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:02.032075 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:02.033051 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:02.033234 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:02.033298 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:02.035416 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:02.035418 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:02.035418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:02.035424 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:02.036824 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:02.037299 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:02.037300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:02.037300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:02.037353 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:02.037566 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:02.037566 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:02.037566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:02.037662 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:02.037710 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:02.037973 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 20:44:02.038043 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:44:02.038189 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:02.029333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38091 (local address 127.0.98.67:41897)
0926 20:44:02.029596 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:02.029636 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:02.030822 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:02.031021 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:02.031024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:02.031269 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:02.031451 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:02.031455 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:02.032151 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:02.032152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:02.032959 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:02.032961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:02.033314 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:02.033315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:02.033409 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:02.033999 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:02.034015 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:02.035338 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:02.036916 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:02.036923 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:02.036925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:02.037224 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:02.037414 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:02.037415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:02.037415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:02.037513 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:02.037715 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:02.037719 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:02.037834 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 20:44:02.037915 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 20:44:02.038034 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":85,"threads_started":1}
I20260926 20:44:02.038712 6426 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38091
I20260926 20:44:02.038808 6426 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:02.038995 6426 heartbeater.cc:511] Master 127.0.98.126:38091 requested a full tablet report, sending...
I20260926 20:44:02.039386 5967 ts_manager.cc:194] Registered new tserver with Master: 9cde82a13f464feaa8401a762593acbb (127.0.98.67:39845)
I20260926 20:44:02.039848 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:41897
I20260926 20:44:02.041531 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/test-admin.keytab.
Sep 26 20:44:02 dist-test-slave-4hd3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:02.056927 393 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 20:44:02.058650 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:02.057865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.0.98.65:38575)
0926 20:44:02.058033 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:02.058038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:02.058046 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:44:02.058073 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:02.058075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:02.058088 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:02.058175 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:02.058550 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34574: BlockingRecv error: recv got EOF from 127.0.0.1:34574 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
W20260926 20:44:02.058831 393 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/krb5cc' of type FILE
Sep 26 20:44:02 dist-test-slave-4hd3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:02.059912 393 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:44:02 dist-test-slave-4hd3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:02.060812 393 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:44:02 dist-test-slave-4hd3 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260926 20:44:02.068710 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:02.061353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34580 (local address 127.0.98.65:38575)
0926 20:44:02.061406 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:02.061407 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:02.061418 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:44:02.061489 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:02.061490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:02.061499 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:02.061549 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:02.061965 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:02.062684 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:02.063426 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:02.063865 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:02.063919 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:02.066021 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:02.066028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:02.066028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:02.066035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:02.067666 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:02.068071 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:02.068074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:02.068074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:02.068129 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:02.068288 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:02.068289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:02.068289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:02.068416 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:02.068517 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:02.068578 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 20:44:02.068638 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260926 20:44:02.070564 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:02.069994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.0.98.65:38575)
0926 20:44:02.070045 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:02.070048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:02.070098 (+ 50us) server_negotiation.cc:407] Connection header received
0926 20:44:02.070168 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:02.070169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:02.070180 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:02.070221 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:02.070516 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34594: BlockingRecv error: recv got EOF from 127.0.0.1:34594 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 20:44:02.070792 393 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/krb5cc' of type FILE
Sep 26 20:44:02 dist-test-slave-4hd3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:02.071818 393 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:44:02 dist-test-slave-4hd3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:02.072729 393 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:44:02 dist-test-slave-4hd3 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260926 20:44:02.079980 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:02.073270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.0.98.65:38575)
0926 20:44:02.073327 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:02.073334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:02.073459 (+ 125us) server_negotiation.cc:407] Connection header received
0926 20:44:02.073488 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:02.073489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:02.073497 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:02.073552 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:02.074016 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:02.074527 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:02.075205 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:02.075409 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:02.075486 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:02.077604 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:02.077606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:02.077606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:02.077612 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:02.078987 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:02.079282 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:02.079283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:02.079283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:02.079330 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:02.079486 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:02.079487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:02.079487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:02.079570 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:02.079639 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:02.079834 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 20:44:02.079919 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 20:44:02.081383 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:02.080927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.0.98.65:38575)
0926 20:44:02.080983 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:02.080985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:02.081011 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:44:02.081059 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:02.081059 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:02.081066 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:02.081109 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:02.081345 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34612: BlockingRecv error: recv got EOF from 127.0.0.1:34612 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 20:44:02.081456 393 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455416624047-393-0/krb5kdc/krb5cc' of type FILE
Sep 26 20:44:02 dist-test-slave-4hd3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:02.082365 393 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:44:02 dist-test-slave-4hd3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:02.083235 393 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:44:02 dist-test-slave-4hd3 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260926 20:44:02.090173 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:02.083775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34614 (local address 127.0.98.65:38575)
0926 20:44:02.083829 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:02.083831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:02.083839 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:44:02.083894 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:02.083895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:02.083904 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:02.083976 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:02.084336 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:02.084805 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:02.085483 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:02.085670 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:02.085744 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:02.087693 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:02.087695 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:02.087695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:02.087702 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:02.089089 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:02.089471 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:02.089473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:02.089473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:02.089527 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:02.089719 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:02.089720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:02.089720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:02.089817 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:02.089915 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:02.090012 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 20:44:02.090082 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:44:02.090768 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 6029
I20260926 20:44:02.095551 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 6163
I20260926 20:44:02.099663 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 6297
I20260926 20:44:02.103768 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 5935
2026-09-26T20:44:02Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1865 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:44:02 dist-test-slave-4hd3 krb5kdc[6446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:44:02 dist-test-slave-4hd3 krb5kdc[6446](info): set up 2 sockets
Sep 26 20:44:02 dist-test-slave-4hd3 krb5kdc[6446](info): commencing operation
krb5kdc: starting...
W20260926 20:44:03.132023 393 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:44:03 dist-test-slave-4hd3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:44:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:44:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.126.keytab.
Entry for principal oryx/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.126.keytab.
I20260926 20:44:03.248697 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:41639
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.126.keytab
--principal=oryx/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:41639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:03.329722 6459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:03.329973 6459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:03.330011 6459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:03.331161 6459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:44:03.331228 6459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:03.331250 6459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:03.331315 6459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:44:03.331413 6459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:44:03.333046 6459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:41639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.126.keytab
--principal=oryx/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:41639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:03.333333 6459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:03.333565 6459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:03.338444 6464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:03.338465 6465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:03.338492 6467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:03.338543 6459 server_base.cc:1034] running on GCE node
I20260926 20:44:03.338896 6459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:03.339192 6459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:03.340368 6459 hybrid_clock.cc:648] HybridClock initialized: now 1790455443340363 us; error 39 us; skew 500 ppm
Sep 26 20:44:03 dist-test-slave-4hd3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:03.342854 6459 init.cc:377] Logged in from keytab as oryx/127.0.98.126@KRBTEST.COM (short username oryx)
I20260926 20:44:03.343322 6459 webserver.cc:460] Webserver started at http://127.0.98.126:46309/ using document root <none> and password file <none>
I20260926 20:44:03.343519 6459 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:03.343559 6459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:03.343696 6459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:03.344776 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "eca7d68fee6a46ec8f522d2a7203f3ef"
format_stamp: "Formatted at 2026-09-26 20:44:03 on dist-test-slave-4hd3"
server_key: "3470fe9286b66dedc8d6ed0475ece9ca"
server_key_iv: "4863177c6f8ff4b6dd809a172f01f87d"
server_key_version: "encryptionkey@0"
I20260926 20:44:03.345079 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "eca7d68fee6a46ec8f522d2a7203f3ef"
format_stamp: "Formatted at 2026-09-26 20:44:03 on dist-test-slave-4hd3"
server_key: "3470fe9286b66dedc8d6ed0475ece9ca"
server_key_iv: "4863177c6f8ff4b6dd809a172f01f87d"
server_key_version: "encryptionkey@0"
I20260926 20:44:03.346347 6459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:44:03.347105 6474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:03.347338 6459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:03.347433 6459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "eca7d68fee6a46ec8f522d2a7203f3ef"
format_stamp: "Formatted at 2026-09-26 20:44:03 on dist-test-slave-4hd3"
server_key: "3470fe9286b66dedc8d6ed0475ece9ca"
server_key_iv: "4863177c6f8ff4b6dd809a172f01f87d"
server_key_version: "encryptionkey@0"
I20260926 20:44:03.347502 6459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:03.369306 6459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:03.371834 6459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:03.371997 6459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:03.375890 6459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:41639
I20260926 20:44:03.376102 6526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:41639 every 8 connection(s)
I20260926 20:44:03.376344 6459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:44:03.377252 6527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:03.379762 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef: Bootstrap starting.
I20260926 20:44:03.380383 6527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:03.380695 6527 log.cc:824] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef: Log is configured to *not* fsync() on all Append() calls
I20260926 20:44:03.381407 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef: No bootstrap required, opened a new log
I20260926 20:44:03.383143 6527 raft_consensus.cc:348] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } }
I20260926 20:44:03.383285 6527 raft_consensus.cc:374] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:03.383309 6527 raft_consensus.cc:729] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eca7d68fee6a46ec8f522d2a7203f3ef, State: Initialized, Role: FOLLOWER
I20260926 20:44:03.383505 6527 consensus_queue.cc:260] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } }
I20260926 20:44:03.383575 6527 raft_consensus.cc:388] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:44:03.383620 6527 raft_consensus.cc:482] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:44:03.383672 6527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:03.384541 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 6459
I20260926 20:44:03.384548 6527 raft_consensus.cc:504] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } }
I20260926 20:44:03.384632 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:44:03.384666 6527 leader_election.cc:302] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eca7d68fee6a46ec8f522d2a7203f3ef; no voters:
I20260926 20:44:03.384863 393 external_mini_cluster.cc:1428] Setting key 1e5ad4b8ac9c47c7e2fcc72e5fc6c3e0
I20260926 20:44:03.384908 6527 leader_election.cc:288] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:44:03.384936 6530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:44:03.385047 6530 raft_consensus.cc:686] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [term 1 LEADER]: Becoming Leader. State: Replica: eca7d68fee6a46ec8f522d2a7203f3ef, State: Running, Role: LEADER
I20260926 20:44:03.385135 6530 consensus_queue.cc:237] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } }
I20260926 20:44:03.385181 6527 sys_catalog.cc:564] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:44:03.385480 6530 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader eca7d68fee6a46ec8f522d2a7203f3ef. Latest consensus state: current_term: 1 leader_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } }
I20260926 20:44:03.385551 6530 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:03.385674 6531 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } }
I20260926 20:44:03.385780 6531 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:03.386171 6536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:44:03.386754 6536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:44:03.388388 6536 catalog_manager.cc:1347] Generated new cluster ID: 6c31914beaf84aacb1952b002eda74f0
I20260926 20:44:03.388447 6536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:44:03 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260926 20:44:03.395849 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:03.385441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.0.98.126:41639)
0926 20:44:03.385916 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:03.385925 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:03.385957 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:44:03.386079 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:03.386086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:03.386146 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:03.387310 (+ 1164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:03.387440 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:03.388689 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:03.389481 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:03.389921 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:03.390055 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:03.392421 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:03.392426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:03.392426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:03.392438 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:03.394170 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:03.394568 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:03.394571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:03.394572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:03.394673 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:03.394885 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:03.394886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:03.394886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:03.395144 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:03.395304 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:03.395423 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 20:44:03.395573 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":73,"threads_started":1}
I20260926 20:44:03.407829 6536 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:44:03.408612 6536 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:44:03.415295 6536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef: Generated new TSK 0
I20260926 20:44:03.415488 6536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.65.keytab.
Entry for principal oryx/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.65.keytab.
I20260926 20:44:03.444741 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41639
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:03.525323 6553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:03.525537 6553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:03.525559 6553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:03.526630 6553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:03.526664 6553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:03.526688 6553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:44:03.528064 6553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:03.528235 6553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:03.528405 6553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:03.533334 6558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:03.533340 6561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:03.533455 6553 server_base.cc:1034] running on GCE node
W20260926 20:44:03.533543 6559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:03.533780 6553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:03.534083 6553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:03.535244 6553 hybrid_clock.cc:648] HybridClock initialized: now 1790455443535244 us; error 40 us; skew 500 ppm
Sep 26 20:44:03 dist-test-slave-4hd3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:03.537838 6553 init.cc:377] Logged in from keytab as oryx/127.0.98.65@KRBTEST.COM (short username oryx)
I20260926 20:44:03.538287 6553 webserver.cc:460] Webserver started at http://127.0.98.65:44703/ using document root <none> and password file <none>
I20260926 20:44:03.538470 6553 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:03.538511 6553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:03.538599 6553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:03.539647 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "789f62973fe7453a87bc5bd18132ba42"
format_stamp: "Formatted at 2026-09-26 20:44:03 on dist-test-slave-4hd3"
server_key: "d5ccbe6b9faa0c43552d6e7048d4c0b2"
server_key_iv: "a6b3f94ace81a25092437b948347246f"
server_key_version: "encryptionkey@0"
I20260926 20:44:03.540035 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "789f62973fe7453a87bc5bd18132ba42"
format_stamp: "Formatted at 2026-09-26 20:44:03 on dist-test-slave-4hd3"
server_key: "d5ccbe6b9faa0c43552d6e7048d4c0b2"
server_key_iv: "a6b3f94ace81a25092437b948347246f"
server_key_version: "encryptionkey@0"
I20260926 20:44:03.541273 6553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:03.541960 6568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:03.542151 6553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:03.542241 6553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "789f62973fe7453a87bc5bd18132ba42"
format_stamp: "Formatted at 2026-09-26 20:44:03 on dist-test-slave-4hd3"
server_key: "d5ccbe6b9faa0c43552d6e7048d4c0b2"
server_key_iv: "a6b3f94ace81a25092437b948347246f"
server_key_version: "encryptionkey@0"
I20260926 20:44:03.542294 6553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:03.565508 6553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:03.567322 6553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:03.567431 6553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:03.567605 6553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:03.567972 6553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:03.568007 6553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:03.568032 6553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:03.568043 6553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:03.573485 6553 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:36235
I20260926 20:44:03.573566 6681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:36235 every 8 connection(s)
I20260926 20:44:03.573882 6553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:44:03.579499 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 6553
I20260926 20:44:03.579593 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:44:03.579946 393 external_mini_cluster.cc:1428] Setting key ffe69441b58026697f07445a62feea98
Sep 26 20:44:03 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260926 20:44:03.584079 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:03.574575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:32871 (local address 127.0.98.126:41639)
0926 20:44:03.574626 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:03.574628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:03.574839 (+ 211us) server_negotiation.cc:407] Connection header received
0926 20:44:03.575371 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:03.575372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:03.575388 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:03.575441 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:03.576654 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:03.577178 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:03.578209 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:03.578415 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:03.578504 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:03.580902 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:03.580904 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:03.580904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:03.580912 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:03.582530 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:03.583039 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:03.583040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:03.583040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:03.583091 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:03.583383 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:03.583383 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:03.583384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:03.583481 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:03.583530 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:03.583892 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 20:44:03.583959 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:44:03.584326 6684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:03.574527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.65:32871)
0926 20:44:03.574687 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:03.574750 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:03.575255 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:03.575513 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:03.575517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:03.576330 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:03.576527 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:03.576534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:03.577301 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:03.577302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:03.578111 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:03.578113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:03.578512 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:03.578513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:03.578640 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:03.579188 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:03.579196 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:03.580799 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:03.582617 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:03.582624 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:03.582627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:03.582962 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:03.583155 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:03.583156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:03.583156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:03.583312 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:03.583543 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:03.583549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:03.583745 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 20:44:03.583864 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 20:44:03.584048 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260926 20:44:03.584810 6682 heartbeater.cc:347] Connected to a master server at 127.0.98.126:41639
I20260926 20:44:03.584915 6682 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:03.585119 6682 heartbeater.cc:511] Master 127.0.98.126:41639 requested a full tablet report, sending...
I20260926 20:44:03.585724 6491 ts_manager.cc:194] Registered new tserver with Master: 789f62973fe7453a87bc5bd18132ba42 (127.0.98.65:36235)
I20260926 20:44:03.586726 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:32871
WARNING: no policy specified for oryx/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.66.keytab.
Entry for principal oryx/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.66.keytab.
I20260926 20:44:03.606261 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41639
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:03.686502 6687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:03.686705 6687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:03.686729 6687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:03.688025 6687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:03.688071 6687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:03.688097 6687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:44:03.689544 6687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:03.689793 6687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:03.690034 6687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:03.695478 6695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:03.695495 6692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:03.695478 6687 server_base.cc:1034] running on GCE node
W20260926 20:44:03.695462 6693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:03.695982 6687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:03.696237 6687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:03.697412 6687 hybrid_clock.cc:648] HybridClock initialized: now 1790455443697398 us; error 48 us; skew 500 ppm
Sep 26 20:44:03 dist-test-slave-4hd3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:03.699932 6687 init.cc:377] Logged in from keytab as oryx/127.0.98.66@KRBTEST.COM (short username oryx)
I20260926 20:44:03.700436 6687 webserver.cc:460] Webserver started at http://127.0.98.66:36153/ using document root <none> and password file <none>
I20260926 20:44:03.700680 6687 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:03.700753 6687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:03.700909 6687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:03.702056 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "a7236e5c160a440aa70f7e343d1623ed"
format_stamp: "Formatted at 2026-09-26 20:44:03 on dist-test-slave-4hd3"
server_key: "b63ffcffa3b9cb8759e02786f6dbdd16"
server_key_iv: "8ab6abd8ae2e75e38b864dafcd7c44be"
server_key_version: "encryptionkey@0"
I20260926 20:44:03.702426 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "a7236e5c160a440aa70f7e343d1623ed"
format_stamp: "Formatted at 2026-09-26 20:44:03 on dist-test-slave-4hd3"
server_key: "b63ffcffa3b9cb8759e02786f6dbdd16"
server_key_iv: "8ab6abd8ae2e75e38b864dafcd7c44be"
server_key_version: "encryptionkey@0"
I20260926 20:44:03.703765 6687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:44:03.704491 6702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:03.704676 6687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:44:03.704772 6687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "a7236e5c160a440aa70f7e343d1623ed"
format_stamp: "Formatted at 2026-09-26 20:44:03 on dist-test-slave-4hd3"
server_key: "b63ffcffa3b9cb8759e02786f6dbdd16"
server_key_iv: "8ab6abd8ae2e75e38b864dafcd7c44be"
server_key_version: "encryptionkey@0"
I20260926 20:44:03.704859 6687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:03.717383 6687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:03.719251 6687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:03.719427 6687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:03.719663 6687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:03.719982 6687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:03.720038 6687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:03.720080 6687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:03.720108 6687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:03.725970 6687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:42795
I20260926 20:44:03.726015 6815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:42795 every 8 connection(s)
I20260926 20:44:03.726336 6687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:44:03.730826 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 6687
I20260926 20:44:03.730957 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:44:03.731146 393 external_mini_cluster.cc:1428] Setting key 9c15d6d58993e1ad73ca0dacdcf1f73c
Sep 26 20:44:03 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260926 20:44:03.736027 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:03.727062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:50961 (local address 127.0.98.126:41639)
0926 20:44:03.727109 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:03.727111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:03.727341 (+ 230us) server_negotiation.cc:407] Connection header received
0926 20:44:03.727948 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:03.727949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:03.727960 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:03.728013 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:03.728975 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:03.729471 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:03.730502 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:03.730697 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:03.730794 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:03.733127 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:03.733128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:03.733128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:03.733135 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:03.734561 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:03.735025 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:03.735026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:03.735027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:03.735084 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:03.735338 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:03.735339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:03.735339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:03.735453 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:03.735525 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:03.735845 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 20:44:03.735937 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:44:03.736227 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:03.727053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.66:50961)
0926 20:44:03.727219 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:03.727254 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:03.727839 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:03.728084 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:03.728088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:03.728672 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:03.728864 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:03.728869 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:03.729564 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:03.729565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:03.730390 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:03.730392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:03.730801 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:03.730802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:03.730960 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:03.731552 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:03.731562 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:03.733022 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:03.734660 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:03.734667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:03.734669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:03.734927 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:03.735150 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:03.735151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:03.735151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:03.735264 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:03.735517 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:03.735523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:03.735665 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 20:44:03.735820 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 20:44:03.735983 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260926 20:44:03.736709 6816 heartbeater.cc:347] Connected to a master server at 127.0.98.126:41639
I20260926 20:44:03.736792 6816 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:03.736976 6816 heartbeater.cc:511] Master 127.0.98.126:41639 requested a full tablet report, sending...
I20260926 20:44:03.737371 6491 ts_manager.cc:194] Registered new tserver with Master: a7236e5c160a440aa70f7e343d1623ed (127.0.98.66:42795)
I20260926 20:44:03.737929 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:50961
WARNING: no policy specified for oryx/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.67.keytab.
Entry for principal oryx/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.67.keytab.
I20260926 20:44:03.754945 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41639
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:03.834285 6821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:03.834492 6821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:03.834515 6821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:03.835685 6821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:03.835750 6821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:03.835839 6821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:44:03.837301 6821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:03.837531 6821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:03.837764 6821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:03.842796 6829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:03.842845 6826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:03.842880 6827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:03.842973 6821 server_base.cc:1034] running on GCE node
I20260926 20:44:03.843405 6821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:03.843647 6821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:03.844817 6821 hybrid_clock.cc:648] HybridClock initialized: now 1790455443844808 us; error 33 us; skew 500 ppm
Sep 26 20:44:03 dist-test-slave-4hd3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:03.847402 6821 init.cc:377] Logged in from keytab as oryx/127.0.98.67@KRBTEST.COM (short username oryx)
I20260926 20:44:03.847936 6821 webserver.cc:460] Webserver started at http://127.0.98.67:39953/ using document root <none> and password file <none>
I20260926 20:44:03.848156 6821 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:03.848224 6821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:03.848379 6821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:03.849510 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "2c15d702d9c842e4b9599c0be7c7c6db"
format_stamp: "Formatted at 2026-09-26 20:44:03 on dist-test-slave-4hd3"
server_key: "39ee4ff32c86e9e82c0e328222287f3f"
server_key_iv: "f1ef4b399de612279058f44629c718a3"
server_key_version: "encryptionkey@0"
I20260926 20:44:03.849946 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "2c15d702d9c842e4b9599c0be7c7c6db"
format_stamp: "Formatted at 2026-09-26 20:44:03 on dist-test-slave-4hd3"
server_key: "39ee4ff32c86e9e82c0e328222287f3f"
server_key_iv: "f1ef4b399de612279058f44629c718a3"
server_key_version: "encryptionkey@0"
I20260926 20:44:03.851333 6821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:44:03.852255 6836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:03.852430 6821 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:44:03.852533 6821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "2c15d702d9c842e4b9599c0be7c7c6db"
format_stamp: "Formatted at 2026-09-26 20:44:03 on dist-test-slave-4hd3"
server_key: "39ee4ff32c86e9e82c0e328222287f3f"
server_key_iv: "f1ef4b399de612279058f44629c718a3"
server_key_version: "encryptionkey@0"
I20260926 20:44:03.852607 6821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:03.867340 6821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:03.869859 6821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:03.870033 6821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:03.870277 6821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:03.870601 6821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:03.870662 6821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:03.870704 6821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:03.870731 6821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:03.876710 6821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:45817
I20260926 20:44:03.876750 6949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:45817 every 8 connection(s)
I20260926 20:44:03.877069 6821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:44:03.879770 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 6821
I20260926 20:44:03.879884 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:44:03.880062 393 external_mini_cluster.cc:1428] Setting key 13c465d906acc3c2062418a808025515
Sep 26 20:44:03 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260926 20:44:03.886580 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:03.877792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:49039 (local address 127.0.98.126:41639)
0926 20:44:03.877838 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:03.877841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:03.878086 (+ 245us) server_negotiation.cc:407] Connection header received
0926 20:44:03.878541 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:03.878542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:03.878552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:03.878607 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:03.879578 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:03.880290 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:03.881507 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:03.881689 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:03.881746 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:03.883908 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:03.883910 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:03.883910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:03.883917 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:03.885142 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:03.885548 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:03.885548 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:03.885549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:03.885595 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:03.885870 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:03.885872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:03.885872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:03.885980 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:03.886058 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:03.886414 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 20:44:03.886496 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:44:03.886746 6952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:03.877792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.67:49039)
0926 20:44:03.877963 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:03.878001 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:03.878465 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:03.878664 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:03.878668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:03.879264 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:03.879472 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:03.879477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:03.880397 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:03.880398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:03.881435 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:03.881437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:03.881779 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:03.881780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:03.881918 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:03.882373 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:03.882381 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:03.883819 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:03.885210 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:03.885215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:03.885217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:03.885489 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:03.885651 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:03.885651 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:03.885652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:03.885746 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:03.886060 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:03.886067 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:03.886190 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 20:44:03.886373 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 20:44:03.886526 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260926 20:44:03.887171 6950 heartbeater.cc:347] Connected to a master server at 127.0.98.126:41639
I20260926 20:44:03.887274 6950 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:03.887472 6950 heartbeater.cc:511] Master 127.0.98.126:41639 requested a full tablet report, sending...
I20260926 20:44:03.887902 6491 ts_manager.cc:194] Registered new tserver with Master: 2c15d702d9c842e4b9599c0be7c7c6db (127.0.98.67:45817)
I20260926 20:44:03.888219 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:49039
I20260926 20:44:03.891619 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.125.keytab.
Entry for principal oryx/127.0.98.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.125.keytab.
I20260926 20:44:03.916204 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:38537
--webserver_interface=127.0.98.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.125.keytab
--principal=oryx/127.0.98.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:41639,127.0.98.125:38537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:03.996440 6955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:03.996665 6955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:03.996702 6955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:03.997838 6955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:44:03.997905 6955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:03.997929 6955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:03.997959 6955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:44:03.997989 6955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:44:03.999405 6955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:41639,127.0.98.125:38537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.125.keytab
--principal=oryx/127.0.98.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:38537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
W20260926 20:44:03.999533 6955 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.126:41639,127.0.98.125:38537'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 20:44:03.999702 6955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:03.999953 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:44:04.004738 6955 server_base.cc:1034] running on GCE node
W20260926 20:44:04.004721 6960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:04.004721 6963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:04.004709 6961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:04.005146 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:04.005393 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:04.006533 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790455444006517 us; error 33 us; skew 500 ppm
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:04.009210 6955 init.cc:377] Logged in from keytab as oryx/127.0.98.125@KRBTEST.COM (short username oryx)
I20260926 20:44:04.009701 6955 webserver.cc:460] Webserver started at http://127.0.98.125:39989/ using document root <none> and password file <none>
I20260926 20:44:04.009943 6955 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:04.010012 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:04.010164 6955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:04.011286 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/data/instance:
uuid: "2ca257ec00324f9ba9742990fd00c469"
format_stamp: "Formatted at 2026-09-26 20:44:04 on dist-test-slave-4hd3"
server_key: "be4504742745ce27ac60f634d0e9bbb1"
server_key_iv: "a35afd7f69fafdbfde6ae3b4bd6f48a5"
server_key_version: "encryptionkey@0"
I20260926 20:44:04.011644 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/wal/instance:
uuid: "2ca257ec00324f9ba9742990fd00c469"
format_stamp: "Formatted at 2026-09-26 20:44:04 on dist-test-slave-4hd3"
server_key: "be4504742745ce27ac60f634d0e9bbb1"
server_key_iv: "a35afd7f69fafdbfde6ae3b4bd6f48a5"
server_key_version: "encryptionkey@0"
I20260926 20:44:04.013006 6955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:44:04.013785 6970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.014005 6955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:04.014104 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/wal
uuid: "2ca257ec00324f9ba9742990fd00c469"
format_stamp: "Formatted at 2026-09-26 20:44:04 on dist-test-slave-4hd3"
server_key: "be4504742745ce27ac60f634d0e9bbb1"
server_key_iv: "a35afd7f69fafdbfde6ae3b4bd6f48a5"
server_key_version: "encryptionkey@0"
I20260926 20:44:04.014187 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:04.028738 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:04.031015 6955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:04.031173 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:04.035303 6955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.125:38537
I20260926 20:44:04.035341 7022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.125:38537 every 8 connection(s)
I20260926 20:44:04.035728 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/data/info.pb
I20260926 20:44:04.036731 7023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:04.038204 7023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:44:04.040767 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 6955
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.125@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
I20260926 20:44:04.048520 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.038858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.0.98.126:41639)
0926 20:44:04.038946 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.038948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.039196 (+ 248us) server_negotiation.cc:407] Connection header received
0926 20:44:04.039748 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.039749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.039762 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.039865 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.040495 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.041281 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.042696 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.042920 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.043010 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.045420 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.045422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.045422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.045430 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.047031 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.047468 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.047469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.047470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.047533 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.047820 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.047821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.047822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.047976 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.048056 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.048342 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.048425 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:44:04.048727 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.038820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:49236)
0926 20:44:04.039058 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.039106 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.039612 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.039948 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.039954 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.040192 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.040395 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.040399 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.041407 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.041408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.042608 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.042611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.043017 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.043018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.043137 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.043638 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.043646 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.045318 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.047106 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.047113 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.047115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.047381 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.047598 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.047599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.047599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.047764 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.048059 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.048065 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.048207 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.048317 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.048480 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":81,"threads_started":1}
I20260926 20:44:04.049348 7023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:44:04.050005 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.041249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.0.98.125:38537)
0926 20:44:04.041429 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.041434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.041459 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:44:04.041503 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.041505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.041521 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.041579 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.042309 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.042985 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.043734 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.044172 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.044267 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.046619 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.046627 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.046628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.046635 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.048413 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.048854 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.048859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.048859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.048927 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.049124 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.049125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.049126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.049295 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.049444 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.049786 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.049890 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.125@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
I20260926 20:44:04.056408 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.049605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:38537 (local address 127.0.0.1:33556)
0926 20:44:04.049650 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.049660 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.049705 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.049911 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.049912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.050094 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.050156 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.050158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.050945 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.050946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.051490 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.051492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.051956 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.051957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.052007 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.052389 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.052394 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.053550 (+ 1156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.055434 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.055435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.055435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.055609 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.055899 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.055900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.055900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.055952 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.056152 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.056153 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.056184 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.056227 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.056318 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 20:44:04.056583 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.049650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33556 (local address 127.0.98.125:38537)
0926 20:44:04.049810 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.049812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.049818 (+ 6us) server_negotiation.cc:407] Connection header received
0926 20:44:04.049834 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.049834 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.049843 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.049906 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.050269 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.050881 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.051566 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.051884 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.051955 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.053638 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.053639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.053640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.053646 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.055336 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.055675 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.055675 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.055675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.055793 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.056009 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.056009 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.056010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.056105 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.056157 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.056249 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.056399 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":143,"thread_start_us":112,"threads_started":1}
I20260926 20:44:04.057807 7023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: Bootstrap starting.
I20260926 20:44:04.058456 7023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:04.058732 7023 log.cc:824] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: Log is configured to *not* fsync() on all Append() calls
I20260926 20:44:04.059432 7023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: No bootstrap required, opened a new log
I20260926 20:44:04.060873 7023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } }
I20260926 20:44:04.061012 7023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:04.061057 7023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca257ec00324f9ba9742990fd00c469, State: Initialized, Role: FOLLOWER
I20260926 20:44:04.061188 7023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } }
I20260926 20:44:04.061436 7030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } } }
I20260926 20:44:04.061501 7023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:44:04.061522 7030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:44:04.063172 7041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:44:04.063232 7041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:44:04.064949 6955 master_runner.cc:428] Detected that this master 2ca257ec00324f9ba9742990fd00c469 is joining an existing cluster
I20260926 20:44:04.064986 6955 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.125:38537
I20260926 20:44:04.073185 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.066107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:38537 (local address 127.0.0.1:33570)
0926 20:44:04.066237 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.066246 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.066285 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.066472 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.066473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.066773 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.066879 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.066880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.067845 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.067846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.068649 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.068650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.069068 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.069068 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.069148 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.069741 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.069745 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.070209 (+ 464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.072134 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.072135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.072135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.072308 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.072507 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.072508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.072508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.072561 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.072802 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.072803 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.072839 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.072944 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.073061 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
I20260926 20:44:04.073189 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.066266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33570 (local address 127.0.98.125:38537)
0926 20:44:04.066362 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.066364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.066379 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:44:04.066397 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.066397 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.066408 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.066468 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.066966 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.067754 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.068738 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.068950 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.069026 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.070301 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.070302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.070303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.070310 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.072034 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.072383 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.072384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.072385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.072434 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.072629 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.072630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.072630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.072739 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.072811 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.072963 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.073110 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":82}
I20260926 20:44:04.073755 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.065973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49246 (local address 127.0.98.126:41639)
0926 20:44:04.066032 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.066035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.066172 (+ 137us) server_negotiation.cc:407] Connection header received
0926 20:44:04.066273 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.066275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.066286 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.066318 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.066982 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.067469 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.068608 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.068916 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.068996 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.070587 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.070588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.070589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.070596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.072349 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.072749 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.072750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.072750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.072804 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.073001 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.073001 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.073002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.073125 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.073167 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.073585 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.073664 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:44:04.074141 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.065898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:49246)
0926 20:44:04.066105 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.066115 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.066187 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.066519 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.066520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.066777 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.066876 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.066878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.067563 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.067564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.068516 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.068518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.069014 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.069015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.069067 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.069584 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.069588 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.070209 (+ 621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.072456 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.072457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.072458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.072664 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.072877 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.072877 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.072877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.072932 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.073214 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.073214 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.073461 (+ 247us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.073933 (+ 472us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.074037 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":4,"thread_start_us":90,"threads_started":1}
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.126@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
I20260926 20:44:04.083305 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.076237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.0.98.125:38537)
0926 20:44:04.076275 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.076277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.076417 (+ 140us) server_negotiation.cc:407] Connection header received
0926 20:44:04.076497 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.076497 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.076505 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.076545 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.077024 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.077450 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.078538 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.078704 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.078757 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.080864 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.080865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.080866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.080880 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.082164 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.082533 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.082535 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.082535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.082591 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.082788 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.082789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.082789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.082895 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.082969 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.083138 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.083213 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:44:04.083340 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.076154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:38537 (local address 127.0.0.1:33572)
0926 20:44:04.076296 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.076350 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.076425 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.076560 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.076562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.076895 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.076951 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.076953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.077523 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.077524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.078456 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.078458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.078777 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.078778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.078858 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.079246 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.079251 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.080777 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.082248 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.082254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.082256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.082458 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.082658 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.082659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.082659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.082724 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.082961 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.082967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.083016 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.083116 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.083226 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260926 20:44:04.083890 6491 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.125:38537: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2ca257ec00324f9ba9742990fd00c469"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.125"
port: 38537
}
attrs {
promote: true
}
}
dest_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef"
cas_config_opid_index: -1
I20260926 20:44:04.084206 6491 consensus_queue.cc:237] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: true } }
I20260926 20:44:04.084682 7051 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2ca257ec00324f9ba9742990fd00c469 (127.0.98.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: true } } }
I20260926 20:44:04.085043 6997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:04.085875 6997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 1 FOLLOWER]: Refusing update from remote peer eca7d68fee6a46ec8f522d2a7203f3ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 20:44:04.085886 7051 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.125:38537
I20260926 20:44:04.085963 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: true } } }
I20260926 20:44:04.086037 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:04.086225 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:44:04.086293 6955 master.cc:537] Master@127.0.98.125:38537 shutting down...
W20260926 20:44:04.086911 6475 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 20:44:04.087390 6475 consensus_peers.cc:487] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef -> Peer 2ca257ec00324f9ba9742990fd00c469 (127.0.98.125:38537): Couldn't send request to peer 2ca257ec00324f9ba9742990fd00c469. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 20:44:04.089242 6955 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:44:04.089336 6955 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:04.089393 6955 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: stopping tablet replica
I20260926 20:44:04.091833 6955 master.cc:552] Master@127.0.98.125:38537 shutdown complete.
I20260926 20:44:04.091886 6955 master_runner.cc:305] Clearing existing system tablet
I20260926 20:44:04.092041 6955 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:04.092100 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:04.092694 6955 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:44:04.093199 7055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.093357 6955 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:04.093421 6955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.093448 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/wal
uuid: "2ca257ec00324f9ba9742990fd00c469"
format_stamp: "Formatted at 2026-09-26 20:44:04 on dist-test-slave-4hd3"
server_key: "be4504742745ce27ac60f634d0e9bbb1"
server_key_iv: "a35afd7f69fafdbfde6ae3b4bd6f48a5"
server_key_version: "encryptionkey@0"
I20260926 20:44:04.094175 6955 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:04.095664 6955 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 20:44:04.095773 6955 log.cc:1196] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 20:44:04.095990 6955 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: Deleting consensus metadata
I20260926 20:44:04.096117 6955 master_runner.cc:315] Copying system tablet from 127.0.98.126:41639
I20260926 20:44:04.096760 6955 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.126:41639
I20260926 20:44:04.103044 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.097060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:49248)
0926 20:44:04.097216 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.097225 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.097279 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.097407 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.097408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.097545 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.097589 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.097591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.098499 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.098500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.099036 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.099037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.099495 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.099496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.099533 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.100063 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.100068 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.100345 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.102207 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.102208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.102208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.102381 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.102570 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.102571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.102571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.102613 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.102823 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.102823 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.102855 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.102905 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.102974 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":73,"threads_started":1}
I20260926 20:44:04.103097 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.097122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49248 (local address 127.0.98.126:41639)
0926 20:44:04.097169 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.097171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.097286 (+ 115us) server_negotiation.cc:407] Connection header received
0926 20:44:04.097339 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.097340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.097351 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.097405 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.097652 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.098418 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.099114 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.099423 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.099492 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.100453 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.100455 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.100455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.100463 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.102132 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.102450 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.102451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.102452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.102511 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.102648 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.102648 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.102649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.102768 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.102845 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.102936 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.103007 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:44:04.103770 6511 tablet_copy_service.cc:140] P eca7d68fee6a46ec8f522d2a7203f3ef: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2ca257ec00324f9ba9742990fd00c469 ({username='oryx', principal='oryx/127.0.98.125@KRBTEST.COM'} at 127.0.0.1:49248)
I20260926 20:44:04.103865 6511 tablet_copy_service.cc:161] P eca7d68fee6a46ec8f522d2a7203f3ef: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2ca257ec00324f9ba9742990fd00c469 at {username='oryx', principal='oryx/127.0.98.125@KRBTEST.COM'} at 127.0.0.1:49248: session id = 2ca257ec00324f9ba9742990fd00c469-00000000000000000000000000000000
I20260926 20:44:04.104641 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 20:44:04.105257 6955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 20:44:04.106442 6955 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: tablet copy: Starting download of 0 data blocks...
I20260926 20:44:04.106585 6955 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: tablet copy: Starting download of 1 WAL segments...
I20260926 20:44:04.107452 6955 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 20:44:04.108359 6511 tablet_copy_service.cc:342] P eca7d68fee6a46ec8f522d2a7203f3ef: Request end of tablet copy session 2ca257ec00324f9ba9742990fd00c469-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.125@KRBTEST.COM'} at 127.0.0.1:49248
I20260926 20:44:04.108422 6511 tablet_copy_service.cc:433] P eca7d68fee6a46ec8f522d2a7203f3ef: ending tablet copy session 2ca257ec00324f9ba9742990fd00c469-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2ca257ec00324f9ba9742990fd00c469
I20260926 20:44:04.110869 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:04.112109 7062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:04.112233 7063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:04.112176 7065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:04.112342 6955 server_base.cc:1034] running on GCE node
I20260926 20:44:04.112493 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:04.112643 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:04.113780 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790455444113772 us; error 23 us; skew 500 ppm
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:04.115039 6955 init.cc:377] Logged in from keytab as oryx/127.0.98.125@KRBTEST.COM (short username oryx)
I20260926 20:44:04.115312 6955 webserver.cc:460] Webserver started at http://127.0.98.125:33969/ using document root <none> and password file <none>
I20260926 20:44:04.115423 6955 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:04.115449 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:04.116127 6955 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:44:04.116604 7072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.116720 6955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.116766 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/wal
uuid: "2ca257ec00324f9ba9742990fd00c469"
format_stamp: "Formatted at 2026-09-26 20:44:04 on dist-test-slave-4hd3"
server_key: "be4504742745ce27ac60f634d0e9bbb1"
server_key_iv: "a35afd7f69fafdbfde6ae3b4bd6f48a5"
server_key_version: "encryptionkey@0"
I20260926 20:44:04.116818 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:04.131453 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:04.133018 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:04.136411 6955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.125:38537
I20260926 20:44:04.136575 7124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.125:38537 every 8 connection(s)
I20260926 20:44:04.136785 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-1/data/info.pb
I20260926 20:44:04.137699 7125 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:44:04.138154 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: Bootstrap starting.
I20260926 20:44:04.140882 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:44:04.141098 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: Bootstrap complete.
I20260926 20:44:04.141278 7125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: true } }
I20260926 20:44:04.141377 7125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ca257ec00324f9ba9742990fd00c469, State: Initialized, Role: LEARNER
I20260926 20:44:04.141420 7125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: true } }
I20260926 20:44:04.141583 7125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:44:04.141577 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: true } } }
I20260926 20:44:04.141701 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:44:04.143042 7139 catalog_manager.cc:1259] Loaded cluster ID: 6c31914beaf84aacb1952b002eda74f0
I20260926 20:44:04.143106 7139 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: loading cluster ID for follower catalog manager: success
I20260926 20:44:04.144310 7139 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: acquiring CA information for follower catalog manager: success
I20260926 20:44:04.144567 7139 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:44:04.158104 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.151569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.0.98.125:38537)
0926 20:44:04.151699 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.151700 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.151707 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:44:04.151767 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.151768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.151779 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.151848 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.152373 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.153029 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.153733 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.153938 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.154009 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.155314 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.155316 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.155316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.155330 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.156929 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.157284 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.157285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.157286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.157356 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.157565 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.157566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.157566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.157676 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.157794 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.157895 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.157975 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.124.keytab.
Entry for principal oryx/127.0.98.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.124.keytab.
I20260926 20:44:04.182274 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:40885
--webserver_interface=127.0.98.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.124.keytab
--principal=oryx/127.0.98.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:41639,127.0.98.125:38537,127.0.98.124:40885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:04.265750 7143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:04.265970 7143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:04.265993 7143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:04.267160 7143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:44:04.267203 7143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:04.267218 7143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:04.267230 7143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:44:04.267243 7143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:44:04.268819 7143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:41639,127.0.98.125:38537,127.0.98.124:40885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.124.keytab
--principal=oryx/127.0.98.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:40885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:04.269078 7143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:04.269327 7143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:44:04.274049 7143 server_base.cc:1034] running on GCE node
W20260926 20:44:04.274030 7149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:04.274106 7148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:04.274320 7151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:04.274573 7143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:04.274914 7143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:04.276072 7143 hybrid_clock.cc:648] HybridClock initialized: now 1790455444276048 us; error 46 us; skew 500 ppm
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:04.278597 7143 init.cc:377] Logged in from keytab as oryx/127.0.98.124@KRBTEST.COM (short username oryx)
I20260926 20:44:04.279057 7143 webserver.cc:460] Webserver started at http://127.0.98.124:36137/ using document root <none> and password file <none>
I20260926 20:44:04.279292 7143 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:04.279361 7143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:04.279510 7143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:04.280614 7143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/data/instance:
uuid: "68fe425de5074e879f7c0e4c90d6b4bd"
format_stamp: "Formatted at 2026-09-26 20:44:04 on dist-test-slave-4hd3"
server_key: "1ad03123ab4b3f180545b683050a19fa"
server_key_iv: "85625da9ccc81ef22f1aa049c5c3b93c"
server_key_version: "encryptionkey@0"
I20260926 20:44:04.280980 7143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/wal/instance:
uuid: "68fe425de5074e879f7c0e4c90d6b4bd"
format_stamp: "Formatted at 2026-09-26 20:44:04 on dist-test-slave-4hd3"
server_key: "1ad03123ab4b3f180545b683050a19fa"
server_key_iv: "85625da9ccc81ef22f1aa049c5c3b93c"
server_key_version: "encryptionkey@0"
I20260926 20:44:04.282312 7143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:44:04.283025 7158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.283196 7143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:44:04.283293 7143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/wal
uuid: "68fe425de5074e879f7c0e4c90d6b4bd"
format_stamp: "Formatted at 2026-09-26 20:44:04 on dist-test-slave-4hd3"
server_key: "1ad03123ab4b3f180545b683050a19fa"
server_key_iv: "85625da9ccc81ef22f1aa049c5c3b93c"
server_key_version: "encryptionkey@0"
I20260926 20:44:04.283372 7143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:04.310338 7143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:04.312786 7143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:04.313001 7143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:04.317804 7143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.124:40885
I20260926 20:44:04.317828 7210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.124:40885 every 8 connection(s)
I20260926 20:44:04.318223 7143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/data/info.pb
I20260926 20:44:04.319240 7211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:04.320827 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.124@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260926 20:44:04.327181 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 7143
I20260926 20:44:04.329878 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.321664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.0.98.126:41639)
0926 20:44:04.321732 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.321738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.321750 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:44:04.322201 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.322202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.322211 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.322277 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.322848 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.323394 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.324429 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.324612 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.324670 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.326805 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.326807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.326807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.326815 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.328431 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.328893 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.328894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.328894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.328949 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.329206 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.329206 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.329207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.329320 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.329374 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.329675 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.329753 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:44:04.329947 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.321487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:49254)
0926 20:44:04.321648 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.321689 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.322113 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.322356 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.322359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.322591 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.322772 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.322775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.323472 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.323473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.324348 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.324350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.324722 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.324723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.324845 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.325329 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.325337 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.326708 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.328520 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.328525 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.328527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.328801 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.329011 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.329012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.329012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.329139 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.329382 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.329386 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.329525 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.329636 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.329769 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260926 20:44:04.330498 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.124@KRBTEST.COM
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.124@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
I20260926 20:44:04.337394 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.327704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37840 (local address 127.0.98.124:40885)
0926 20:44:04.327896 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.327901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.327923 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:44:04.327944 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.327946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.327960 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.328041 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.328559 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.329540 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.330559 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.330984 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.331047 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.333652 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.333659 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.333660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.333667 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.336021 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.336420 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.336422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.336422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.336489 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.336713 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.336714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.336714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.336966 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.337092 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.337194 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.337271 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260926 20:44:04.337854 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.330823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.0.98.125:38537)
0926 20:44:04.330863 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.330864 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.330873 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:44:04.330939 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.330940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.330950 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.330992 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.331292 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.332088 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.332950 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.333138 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.333220 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.335004 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.335006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.335006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.335013 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.336604 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.337039 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.337040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.337040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.337097 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.337300 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.337301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.337301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.337434 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.337526 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.337722 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.337794 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:44:04.337881 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.330771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:38537 (local address 127.0.0.1:33598)
0926 20:44:04.330818 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.330835 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.330876 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.330996 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.330997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.331144 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.331188 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.331189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.332185 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.332185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.332814 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.332816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.333220 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.333221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.333290 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.333713 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.333717 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.334915 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.336710 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.336711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.336711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.336952 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.337166 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.337167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.337167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.337226 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.337560 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.337561 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.337605 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.337676 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.337760 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 20:44:04.338204 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.124@KRBTEST.COM for oryx/127.0.98.124@KRBTEST.COM
I20260926 20:44:04.344214 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.338446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40885 (local address 127.0.0.1:37854)
0926 20:44:04.338491 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.338502 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.338537 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.338646 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.338646 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.338744 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.338769 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.338770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.339317 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.339317 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.339877 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.339879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.340176 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.340177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.340210 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.340505 (+ 295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.340509 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.341612 (+ 1103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.343299 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.343300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.343300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.343451 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.343636 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.343636 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.343637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.343693 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.343975 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.343975 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.344010 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.344064 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.344140 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 20:44:04.344307 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.338512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37854 (local address 127.0.98.124:40885)
0926 20:44:04.338544 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.338546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.338577 (+ 31us) server_negotiation.cc:407] Connection header received
0926 20:44:04.338598 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.338598 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.338606 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.338642 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.338826 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.339250 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.339941 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.340116 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.340167 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.341681 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.341682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.341682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.341688 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.343226 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.343513 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.343514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.343514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.343559 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.343824 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.343824 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.343825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.343917 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.343966 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.344092 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.344182 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":15}
I20260926 20:44:04.345715 7211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: Bootstrap starting.
I20260926 20:44:04.346481 7211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:04.346699 7211 log.cc:824] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: Log is configured to *not* fsync() on all Append() calls
I20260926 20:44:04.347339 7211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: No bootstrap required, opened a new log
I20260926 20:44:04.348814 7211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } }
I20260926 20:44:04.348951 7211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:04.348973 7211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68fe425de5074e879f7c0e4c90d6b4bd, State: Initialized, Role: FOLLOWER
I20260926 20:44:04.349095 7211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } }
I20260926 20:44:04.349344 7211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:44:04.349336 7216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } } }
I20260926 20:44:04.349437 7216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:44:04.351161 7227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:44:04.351285 7227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:44:04.353535 7143 master_runner.cc:428] Detected that this master 68fe425de5074e879f7c0e4c90d6b4bd is joining an existing cluster
I20260926 20:44:04.353585 7143 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.124:40885
I20260926 20:44:04.361446 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.354764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37864 (local address 127.0.98.124:40885)
0926 20:44:04.354800 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.354802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.354920 (+ 118us) server_negotiation.cc:407] Connection header received
0926 20:44:04.355007 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.355008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.355017 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.355043 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.355475 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.356226 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.357244 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.357530 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.357568 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.358732 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.358733 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.358733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.358739 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.360310 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.360680 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.360681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.360681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.360741 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.360922 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.360923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.360923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.361046 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.361118 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.361263 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.361365 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":16}
I20260926 20:44:04.361575 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.354692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40885 (local address 127.0.0.1:37864)
0926 20:44:04.354860 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.354870 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.354943 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.355094 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.355095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.355316 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.355391 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.355393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.356313 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.356313 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.357154 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.357156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.357638 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.357639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.357694 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.358185 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.358190 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.358642 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.360424 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.360425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.360425 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.360604 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.360810 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.360810 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.360810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.360861 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.361115 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.361116 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.361163 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.361219 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.361295 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
I20260926 20:44:04.376027 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.354983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33602 (local address 127.0.98.125:38537)
0926 20:44:04.355045 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.355048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.355271 (+ 223us) server_negotiation.cc:407] Connection header received
0926 20:44:04.355368 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.355369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.355379 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.355410 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.355909 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.356683 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.358216 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.358502 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.358542 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.373252 (+ 14710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.373254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.373254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.373262 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.374669 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.375156 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.375157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.375157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.375217 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.375412 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.375413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.375413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.375552 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.375621 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.375841 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.375928 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:44:04.376034 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.355072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49266 (local address 127.0.98.126:41639)
0926 20:44:04.355139 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.355141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.355164 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:44:04.355244 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.355244 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.355255 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.355309 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.355847 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.356627 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.358242 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.358544 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.358584 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.372965 (+ 14381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.372967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.372968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.372977 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.374673 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.375192 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.375193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.375194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.375253 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.375472 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.375473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.375473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.375612 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.375671 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.375881 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.375961 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:44:04.376376 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.354936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:38537 (local address 127.0.0.1:33602)
0926 20:44:04.355208 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.355217 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.355295 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.355471 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.355471 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.355684 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.355824 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.355826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.356797 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.356797 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.358112 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.358114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.358619 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.358620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.358685 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.372204 (+ 13519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.372210 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.372825 (+ 615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.374776 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.374777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.374778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.375081 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.375283 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.375284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.375284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.375340 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.375624 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.375625 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.375683 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.376121 (+ 438us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.376236 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":22,"thread_start_us":81,"threads_started":1}
I20260926 20:44:04.376605 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.354695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:49266)
0926 20:44:04.355106 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.355116 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.355157 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.355376 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.355377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.355632 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.355702 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.355704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.356722 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.356722 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.358154 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.358155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.358649 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.358650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.358712 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.372379 (+ 13667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.372384 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.372825 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.374775 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.374775 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.374776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.375089 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.375324 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.375325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.375325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.375391 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.375675 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.375676 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.375749 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.376157 (+ 408us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.376301 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":98,"thread_start_us":123,"threads_started":1}
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.126@KRBTEST.COM for oryx/127.0.98.124@KRBTEST.COM
I20260926 20:44:04.385319 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.378676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40885 (local address 127.0.0.1:37866)
0926 20:44:04.378727 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.378736 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.378772 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.378922 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.378922 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.379078 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.379123 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.379125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.379855 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.379856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.380676 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.380678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.380995 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.380996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.381067 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.381397 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.381401 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.382663 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.384251 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.384252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.384253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.384430 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.384642 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.384643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.384644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.384706 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.385005 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.385006 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.385048 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.385137 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.385229 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 20:44:04.385329 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.378737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37866 (local address 127.0.98.124:40885)
0926 20:44:04.378779 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.378781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.378793 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:44:04.378831 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.378832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.378845 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.378913 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.379199 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.379774 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.380758 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.380930 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.380974 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.382750 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.382751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.382751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.382758 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.384151 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.384516 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.384517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.384518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.384568 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.384769 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.384770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.384771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.384933 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.385013 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.385158 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.385227 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:44:04.385839 6491 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.124:40885: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.124"
port: 40885
}
attrs {
promote: true
}
}
dest_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef"
cas_config_opid_index: 5
I20260926 20:44:04.386072 6491 consensus_queue.cc:237] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: true } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: true } }
I20260926 20:44:04.386583 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 68fe425de5074e879f7c0e4c90d6b4bd (127.0.98.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: true } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: true } } }
I20260926 20:44:04.386668 7185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:04.387374 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.124:40885
I20260926 20:44:04.387458 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: true } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: true } } }
I20260926 20:44:04.387535 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:04.387609 7185 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 1 FOLLOWER]: Refusing update from remote peer eca7d68fee6a46ec8f522d2a7203f3ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 20:44:04.387971 7143 master.cc:537] Master@127.0.98.124:40885 shutting down...
I20260926 20:44:04.387998 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 20:44:04.390115 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef -> Peer 68fe425de5074e879f7c0e4c90d6b4bd (127.0.98.124:40885): Couldn't send request to peer 68fe425de5074e879f7c0e4c90d6b4bd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 20:44:04.390820 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.386475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:38537 (local address 127.0.0.1:33610)
0926 20:44:04.387635 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.387645 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.387694 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.387882 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.387883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.387884 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:04.387916 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.387918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.388791 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.388792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.389974 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.389976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.390595 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.390596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.390650 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.390667 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.390726 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.390760 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1146}
I20260926 20:44:04.391043 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.386552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.0.98.125:38537)
0926 20:44:04.386590 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.386592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.387683 (+ 1091us) server_negotiation.cc:407] Connection header received
0926 20:44:04.387787 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.387788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.387808 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.387870 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:04.387989 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.388709 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.390061 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.390501 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.390597 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.390668 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.390784 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.390820 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:44:04.391258 7099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 1 LEARNER]: Refusing update from remote peer eca7d68fee6a46ec8f522d2a7203f3ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 20:44:04.391479 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 20:44:04.392074 7099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 68fe425de5074e879f7c0e4c90d6b4bd (127.0.98.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: true } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: true } } }
I20260926 20:44:04.393142 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: true } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: true } } }
I20260926 20:44:04.393249 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:44:04.393334 7051 raft_consensus.cc:1050] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef: attempting to promote NON_VOTER 2ca257ec00324f9ba9742990fd00c469 to VOTER
I20260926 20:44:04.393115 7237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eca7d68fee6a46ec8f522d2a7203f3ef. Latest consensus state: current_term: 1 leader_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: true } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: true } } }
I20260926 20:44:04.393501 7237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:44:04.393523 7051 consensus_queue.cc:237] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: true } }
I20260926 20:44:04.394135 7099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 1 LEARNER]: Refusing update from remote peer eca7d68fee6a46ec8f522d2a7203f3ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260926 20:44:04.394295 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef -> Peer 68fe425de5074e879f7c0e4c90d6b4bd (127.0.98.124:40885): Couldn't send request to peer 68fe425de5074e879f7c0e4c90d6b4bd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 20:44:04.394301 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 20:44:04.395000 7143 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:44:04.395092 7143 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:04.395045 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2ca257ec00324f9ba9742990fd00c469 (127.0.98.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: true } } }
I20260926 20:44:04.395145 7143 tablet_replica.cc:330] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: stopping tablet replica
I20260926 20:44:04.395884 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: true } } }
I20260926 20:44:04.395958 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:04.396004 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: true } } }
I20260926 20:44:04.396051 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:04.396297 7099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2ca257ec00324f9ba9742990fd00c469 (127.0.98.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: true } } }
W20260926 20:44:04.396502 6444 connection.cc:535] client connection to 127.0.98.124:40885 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:44:04.397310 7237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: true } } }
I20260926 20:44:04.397401 7237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:44:04.408259 7143 master.cc:552] Master@127.0.98.124:40885 shutdown complete.
I20260926 20:44:04.408558 7143 master_runner.cc:305] Clearing existing system tablet
I20260926 20:44:04.408735 7143 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:04.408800 7143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:04.409412 7143 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:44:04.409934 7240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.410092 7143 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:04.410161 7143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.410187 7143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/wal
uuid: "68fe425de5074e879f7c0e4c90d6b4bd"
format_stamp: "Formatted at 2026-09-26 20:44:04 on dist-test-slave-4hd3"
server_key: "1ad03123ab4b3f180545b683050a19fa"
server_key_iv: "85625da9ccc81ef22f1aa049c5c3b93c"
server_key_version: "encryptionkey@0"
I20260926 20:44:04.410935 7143 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:04.412413 7143 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 20:44:04.412482 7143 log.cc:1196] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 20:44:04.412669 7143 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: Deleting consensus metadata
I20260926 20:44:04.412978 7143 master_runner.cc:315] Copying system tablet from 127.0.98.126:41639
I20260926 20:44:04.413573 7143 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.126:41639
I20260926 20:44:04.420454 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.413927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.0.98.126:41639)
0926 20:44:04.413978 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.413980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.414058 (+ 78us) server_negotiation.cc:407] Connection header received
0926 20:44:04.414115 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.414116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.414124 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.414172 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.414589 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.415126 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.416241 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.416430 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.416488 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.418004 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.418005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.418006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.418012 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.419374 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.419783 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.419784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.419785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.419849 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.420038 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.420039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.420039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.420135 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.420183 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.420340 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.420393 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 20:44:04.420508 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.413841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:49282)
0926 20:44:04.413975 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.413987 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.414055 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.414177 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.414178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.414379 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.414448 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.414450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.415209 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.415210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.416145 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.416147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.416502 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.416503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.416570 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.417494 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.417499 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.417906 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.419456 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.419457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.419458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.419678 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.419914 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.419914 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.419915 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.419978 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.420190 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.420191 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.420235 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.420302 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.420401 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":94,"threads_started":1}
I20260926 20:44:04.420694 6511 tablet_copy_service.cc:140] P eca7d68fee6a46ec8f522d2a7203f3ef: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 68fe425de5074e879f7c0e4c90d6b4bd ({username='oryx', principal='oryx/127.0.98.124@KRBTEST.COM'} at 127.0.0.1:49282)
I20260926 20:44:04.420744 6511 tablet_copy_service.cc:161] P eca7d68fee6a46ec8f522d2a7203f3ef: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 68fe425de5074e879f7c0e4c90d6b4bd at {username='oryx', principal='oryx/127.0.98.124@KRBTEST.COM'} at 127.0.0.1:49282: session id = 68fe425de5074e879f7c0e4c90d6b4bd-00000000000000000000000000000000
I20260926 20:44:04.421183 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 20:44:04.421635 7143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 20:44:04.422513 7143 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: tablet copy: Starting download of 0 data blocks...
I20260926 20:44:04.422668 7143 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: tablet copy: Starting download of 1 WAL segments...
I20260926 20:44:04.423445 7143 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 20:44:04.424278 6511 tablet_copy_service.cc:342] P eca7d68fee6a46ec8f522d2a7203f3ef: Request end of tablet copy session 68fe425de5074e879f7c0e4c90d6b4bd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.124@KRBTEST.COM'} at 127.0.0.1:49282
I20260926 20:44:04.424332 6511 tablet_copy_service.cc:433] P eca7d68fee6a46ec8f522d2a7203f3ef: ending tablet copy session 68fe425de5074e879f7c0e4c90d6b4bd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 68fe425de5074e879f7c0e4c90d6b4bd
I20260926 20:44:04.426715 7143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:04.427765 7247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:04.427884 7248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:04.427987 7143 server_base.cc:1034] running on GCE node
W20260926 20:44:04.427896 7250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:04.428139 7143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:04.428280 7143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:04.429425 7143 hybrid_clock.cc:648] HybridClock initialized: now 1790455444429416 us; error 24 us; skew 500 ppm
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:04.430696 7143 init.cc:377] Logged in from keytab as oryx/127.0.98.124@KRBTEST.COM (short username oryx)
I20260926 20:44:04.430974 7143 webserver.cc:460] Webserver started at http://127.0.98.124:39573/ using document root <none> and password file <none>
I20260926 20:44:04.431075 7143 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:04.431100 7143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:04.431679 7143 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.432217 7257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.432348 7143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.432394 7143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/wal
uuid: "68fe425de5074e879f7c0e4c90d6b4bd"
format_stamp: "Formatted at 2026-09-26 20:44:04 on dist-test-slave-4hd3"
server_key: "1ad03123ab4b3f180545b683050a19fa"
server_key_iv: "85625da9ccc81ef22f1aa049c5c3b93c"
server_key_version: "encryptionkey@0"
I20260926 20:44:04.432451 7143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:04.446203 7143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:04.448007 7143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:04.451565 7143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.124:40885
I20260926 20:44:04.451772 7309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.124:40885 every 8 connection(s)
I20260926 20:44:04.451975 7143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/master-2/data/info.pb
I20260926 20:44:04.452703 7310 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:44:04.453137 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: Bootstrap starting.
I20260926 20:44:04.455513 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:44:04.455792 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: Bootstrap complete.
I20260926 20:44:04.456010 7310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: true } }
I20260926 20:44:04.456095 7310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 68fe425de5074e879f7c0e4c90d6b4bd, State: Initialized, Role: LEARNER
I20260926 20:44:04.456123 7310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: true } }
I20260926 20:44:04.456326 7310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:44:04.456326 7313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: true } } }
I20260926 20:44:04.456449 7313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [sys.catalog]: This master's current role is: LEARNER
I20260926 20:44:04.457674 7324 catalog_manager.cc:1259] Loaded cluster ID: 6c31914beaf84aacb1952b002eda74f0
I20260926 20:44:04.457708 7324 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: loading cluster ID for follower catalog manager: success
I20260926 20:44:04.458662 7324 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: acquiring CA information for follower catalog manager: success
I20260926 20:44:04.458885 7324 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:44:04.504202 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.497888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37884 (local address 127.0.98.124:40885)
0926 20:44:04.498041 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.498043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.498050 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:44:04.498074 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.498075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.498085 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.498159 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.498764 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.499415 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.500152 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.500354 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.500443 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.501684 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.501686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.501686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.501699 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.503001 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.503359 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.503360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.503361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.503419 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.503614 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.503615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.503616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.503780 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.503900 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.504006 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.504089 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260926 20:44:04.504781 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 6553
I20260926 20:44:04.509730 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:36235
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:41639
--webserver_port=44703
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.126:41639,127.0.98.125:38537
--tserver_master_addrs=127.0.98.126:41639,127.0.98.125:38537,127.0.98.124:40885 with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:04.591434 7326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:04.591663 7326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:04.591688 7326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:04.592962 7326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:04.593011 7326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:04.593039 7326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:44:04.594893 7326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:36235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=44703
--tserver_master_addrs=127.0.98.126:41639,127.0.98.125:38537,127.0.98.124:40885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:04.595178 7326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:04.595418 7326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:04.601169 7331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:04.601161 7334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:04.601248 7326 server_base.cc:1034] running on GCE node
W20260926 20:44:04.601161 7332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:04.601596 7326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:04.601833 7326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:04.603003 7326 hybrid_clock.cc:648] HybridClock initialized: now 1790455444602988 us; error 31 us; skew 500 ppm
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:04.605520 7326 init.cc:377] Logged in from keytab as oryx/127.0.98.65@KRBTEST.COM (short username oryx)
I20260926 20:44:04.605953 7326 webserver.cc:460] Webserver started at http://127.0.98.65:44703/ using document root <none> and password file <none>
I20260926 20:44:04.606187 7326 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:04.606274 7326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:04.607686 7326 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:04.608381 7341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.608574 7326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:44:04.608672 7326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "789f62973fe7453a87bc5bd18132ba42"
format_stamp: "Formatted at 2026-09-26 20:44:03 on dist-test-slave-4hd3"
server_key: "d5ccbe6b9faa0c43552d6e7048d4c0b2"
server_key_iv: "a6b3f94ace81a25092437b948347246f"
server_key_version: "encryptionkey@0"
I20260926 20:44:04.609052 7326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:04.620961 7326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:04.622828 7326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:04.622983 7326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:04.623216 7326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:04.623561 7326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:04.623610 7326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.623639 7326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:04.623656 7326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.629474 7326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:36235
I20260926 20:44:04.629537 7454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:36235 every 8 connection(s)
I20260926 20:44:04.629889 7326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:44:04.634362 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 7326
I20260926 20:44:04.634529 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 6687
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
I20260926 20:44:04.640990 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:42795
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:41639
--webserver_port=36153
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.126:41639,127.0.98.125:38537
--tserver_master_addrs=127.0.98.126:41639,127.0.98.125:38537,127.0.98.124:40885 with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.124@KRBTEST.COM
I20260926 20:44:04.643567 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.630637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:58069 (local address 127.0.98.125:38537)
0926 20:44:04.630687 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.630689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.631072 (+ 383us) server_negotiation.cc:407] Connection header received
0926 20:44:04.632360 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.632360 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.632373 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.632431 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.633135 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.633925 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.635343 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.635639 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.635678 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.638001 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.638002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.638002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.638010 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.639432 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.639930 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.639932 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.639932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.639993 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.640465 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.640466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.640466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.640619 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.640692 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.643377 (+ 2685us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.643469 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260926 20:44:04.644176 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.630589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:38537 (local address 127.0.98.65:58069)
0926 20:44:04.630954 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.630987 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.632254 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.632511 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.632519 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.632796 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.633041 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.633046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.634060 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.634061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.635234 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.635236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.635801 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.635802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.635937 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.636469 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.636489 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.637901 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.639536 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.639543 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.639545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.639871 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.640287 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.640287 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.640288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.640383 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.640704 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.642771 (+ 2067us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.643195 (+ 424us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.643738 (+ 543us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.643926 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":27,"thread_start_us":293,"threads_started":1}
I20260926 20:44:04.644948 7457 heartbeater.cc:347] Connected to a master server at 127.0.98.125:38537
I20260926 20:44:04.645058 7457 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:04.645339 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.630590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40885 (local address 127.0.98.65:42203)
0926 20:44:04.631233 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.631243 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.632257 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.632515 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.632521 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.632881 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.633042 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.633048 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.634060 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.634061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.635799 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.635801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.640080 (+ 4279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.640081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.640158 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.640574 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.640578 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.641966 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.643971 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.643972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.643973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.644187 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.644598 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.644599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.644599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.644660 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.644946 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.644947 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.644992 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.645062 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.645171 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":102,"thread_start_us":291,"threads_started":1}
I20260926 20:44:04.645515 7457 heartbeater.cc:511] Master 127.0.98.125:38537 requested a full tablet report, sending...
I20260926 20:44:04.645989 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.630613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:42203 (local address 127.0.98.124:40885)
0926 20:44:04.630751 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.630753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.631298 (+ 545us) server_negotiation.cc:407] Connection header received
0926 20:44:04.632397 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.632399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.632414 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.632444 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.633174 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.633933 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.639774 (+ 5841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.640012 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.640063 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.642059 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.642061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.642061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.642068 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.643901 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.644460 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.644462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.644462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.644522 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.644721 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.644721 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.644722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.644857 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.644919 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.645752 (+ 833us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.645857 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260926 20:44:04.646032 7089 ts_manager.cc:194] Registered new tserver with Master: 789f62973fe7453a87bc5bd18132ba42 (127.0.98.65:36235)
I20260926 20:44:04.646493 7455 heartbeater.cc:347] Connected to a master server at 127.0.98.124:40885
I20260926 20:44:04.646559 7455 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:04.646624 7455 heartbeater.cc:511] Master 127.0.98.124:40885 requested a full tablet report, sending...
I20260926 20:44:04.647011 7274 ts_manager.cc:194] Registered new tserver with Master: 789f62973fe7453a87bc5bd18132ba42 (127.0.98.65:36235)
I20260926 20:44:04.647791 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.631224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:58197 (local address 127.0.98.126:41639)
0926 20:44:04.631270 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.631272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.631343 (+ 71us) server_negotiation.cc:407] Connection header received
0926 20:44:04.632379 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.632381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.632394 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.632428 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.633154 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.633974 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.641822 (+ 7848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.642146 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.642193 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.644420 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.644422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.644422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.644432 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.646114 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.646948 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.646950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.646950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.647020 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.647274 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.647275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.647275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.647395 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.647472 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.647612 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.647685 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:44:04.647797 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.631121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.65:58197)
0926 20:44:04.631292 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.631300 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.632257 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.632515 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.632521 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.632893 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.633042 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.633048 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.640784 (+ 7736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.640785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.641723 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.641726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.642278 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.642279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.642392 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.642789 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.642794 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.644304 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.646203 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.646204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.646205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.646850 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.647138 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.647139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.647139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.647205 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.647463 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.647464 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.647503 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.647562 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.647668 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":4,"thread_start_us":62,"threads_started":1}
I20260926 20:44:04.648317 7456 heartbeater.cc:347] Connected to a master server at 127.0.98.126:41639
I20260926 20:44:04.648368 7456 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:04.648437 7456 heartbeater.cc:511] Master 127.0.98.126:41639 requested a full tablet report, sending...
I20260926 20:44:04.648690 6491 ts_manager.cc:194] Re-registered known tserver with Master: 789f62973fe7453a87bc5bd18132ba42 (127.0.98.65:36235)
I20260926 20:44:04.649128 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:58197
W20260926 20:44:04.724042 7462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:04.724273 7462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:04.724308 7462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:04.725447 7462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:04.725520 7462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:04.725564 7462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:44:04.727043 7462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:42795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=36153
--tserver_master_addrs=127.0.98.126:41639,127.0.98.125:38537,127.0.98.124:40885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:04.727310 7462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:04.727541 7462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:04.733057 7467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:04.733011 7470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:04.733011 7468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:04.733412 7462 server_base.cc:1034] running on GCE node
I20260926 20:44:04.733620 7462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:04.733847 7462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:04.735006 7462 hybrid_clock.cc:648] HybridClock initialized: now 1790455444734990 us; error 33 us; skew 500 ppm
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:04.737708 7462 init.cc:377] Logged in from keytab as oryx/127.0.98.66@KRBTEST.COM (short username oryx)
I20260926 20:44:04.738162 7462 webserver.cc:460] Webserver started at http://127.0.98.66:36153/ using document root <none> and password file <none>
I20260926 20:44:04.738416 7462 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:04.738502 7462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:04.739904 7462 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:04.740525 7477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.740712 7462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:44:04.740814 7462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "a7236e5c160a440aa70f7e343d1623ed"
format_stamp: "Formatted at 2026-09-26 20:44:03 on dist-test-slave-4hd3"
server_key: "b63ffcffa3b9cb8759e02786f6dbdd16"
server_key_iv: "8ab6abd8ae2e75e38b864dafcd7c44be"
server_key_version: "encryptionkey@0"
I20260926 20:44:04.741161 7462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:04.758991 7462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:04.761476 7462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:04.761593 7462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:04.761777 7462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:04.762113 7462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:04.762151 7462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.762174 7462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:04.762187 7462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.767867 7462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:42795
I20260926 20:44:04.767925 7590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:42795 every 8 connection(s)
I20260926 20:44:04.768255 7462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.124@KRBTEST.COM
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
I20260926 20:44:04.776901 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 7462
I20260926 20:44:04.777061 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 6821
I20260926 20:44:04.779590 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.769265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:54983 (local address 127.0.98.124:40885)
0926 20:44:04.769318 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.769320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.769331 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:44:04.770054 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.770055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.770066 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.770139 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.770732 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.771499 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.772632 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.772913 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.772967 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.776200 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.776201 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.776202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.776209 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.778135 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.778545 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.778546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.778547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.778613 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.778876 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.778877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.778877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.778993 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.779065 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.779411 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.779490 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:44:04.779798 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.768992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40885 (local address 127.0.98.66:54983)
0926 20:44:04.769158 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.769203 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.769877 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.770160 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.770166 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.770440 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.770640 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.770644 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.771584 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.771585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.772548 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.772550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.772982 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.772983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.773036 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.773622 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.773633 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.776093 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.778228 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.778235 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.778237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.778466 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.778681 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.778682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.778682 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.778796 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.779071 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.779077 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.779251 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.779382 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.779545 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":3,"thread_start_us":89,"threads_started":1}
I20260926 20:44:04.780411 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.769020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:38229 (local address 127.0.98.126:41639)
0926 20:44:04.769088 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.769090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.769468 (+ 378us) server_negotiation.cc:407] Connection header received
0926 20:44:04.770016 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.770018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.770030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.770081 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.770820 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.771565 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.772492 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.772750 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.772814 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.775816 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.775817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.775818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.775827 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.778816 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.779215 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.779216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.779216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.779277 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.779510 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.779511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.779512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.779653 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.779693 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.780237 (+ 544us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.780325 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:44:04.780408 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.769013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.66:38229)
0926 20:44:04.769385 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.769396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.769879 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.770557 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.770558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.770699 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.770756 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.770757 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.771644 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.771645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.772386 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.772388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.772832 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.772833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.772981 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.773996 (+ 1015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.774001 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.775673 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.778905 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.778906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.778907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.779140 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.779355 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.779356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.779357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.779427 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.779985 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.779985 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.780046 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.780137 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.780247 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":23,"thread_start_us":91,"threads_started":1}
I20260926 20:44:04.780548 7591 heartbeater.cc:347] Connected to a master server at 127.0.98.124:40885
I20260926 20:44:04.780650 7591 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:04.780844 7591 heartbeater.cc:511] Master 127.0.98.124:40885 requested a full tablet report, sending...
I20260926 20:44:04.781332 7274 ts_manager.cc:194] Registered new tserver with Master: a7236e5c160a440aa70f7e343d1623ed (127.0.98.66:42795)
I20260926 20:44:04.782747 7592 heartbeater.cc:347] Connected to a master server at 127.0.98.126:41639
I20260926 20:44:04.782804 7592 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:04.782886 7592 heartbeater.cc:511] Master 127.0.98.126:41639 requested a full tablet report, sending...
I20260926 20:44:04.783118 6491 ts_manager.cc:194] Re-registered known tserver with Master: a7236e5c160a440aa70f7e343d1623ed (127.0.98.66:42795)
I20260926 20:44:04.783509 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:38229
I20260926 20:44:04.784607 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:45817
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:41639
--webserver_port=39953
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.126:41639,127.0.98.125:38537
--tserver_master_addrs=127.0.98.126:41639,127.0.98.125:38537,127.0.98.124:40885 with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:44:04.785434 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.769450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:58533 (local address 127.0.98.125:38537)
0926 20:44:04.769494 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.769496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.769566 (+ 70us) server_negotiation.cc:407] Connection header received
0926 20:44:04.770141 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.770142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.770153 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.770208 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.770750 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.771301 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.773018 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.773197 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.773274 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.776199 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.776200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.776201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.776209 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.784226 (+ 8017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.784642 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.784643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.784643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.784693 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.784904 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.784905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.784905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.785022 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.785098 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.785248 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.785325 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:44:04.785638 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.769271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:38537 (local address 127.0.98.66:58533)
0926 20:44:04.769455 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.769464 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.769884 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.770277 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.770278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.770530 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.770642 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.770647 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.771411 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.771411 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.772548 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.772550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.773273 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.773274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.773339 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.773748 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.773752 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.776119 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.784298 (+ 8179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.784299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.784300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.784561 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.784774 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.784775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.784776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.784841 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.785091 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.785092 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.785133 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.785224 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.785328 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":2,"thread_start_us":65,"threads_started":1}
I20260926 20:44:04.786166 7593 heartbeater.cc:347] Connected to a master server at 127.0.98.125:38537
I20260926 20:44:04.786211 7593 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:04.786288 7593 heartbeater.cc:511] Master 127.0.98.125:38537 requested a full tablet report, sending...
I20260926 20:44:04.786639 7089 ts_manager.cc:194] Registered new tserver with Master: a7236e5c160a440aa70f7e343d1623ed (127.0.98.66:42795)
W20260926 20:44:04.865558 7598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:04.865775 7598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:04.865800 7598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:04.866981 7598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:04.867025 7598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:04.867053 7598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:44:04.868546 7598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:45817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=39953
--tserver_master_addrs=127.0.98.126:41639,127.0.98.125:38537,127.0.98.124:40885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:04.868781 7598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:04.868999 7598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:04.874121 7604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:04.874115 7606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:04.874131 7598 server_base.cc:1034] running on GCE node
W20260926 20:44:04.874115 7603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:04.874486 7598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:04.874682 7598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:04.875787 7598 hybrid_clock.cc:648] HybridClock initialized: now 1790455444875775 us; error 27 us; skew 500 ppm
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:04.878279 7598 init.cc:377] Logged in from keytab as oryx/127.0.98.67@KRBTEST.COM (short username oryx)
I20260926 20:44:04.878705 7598 webserver.cc:460] Webserver started at http://127.0.98.67:39953/ using document root <none> and password file <none>
I20260926 20:44:04.878921 7598 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:04.878984 7598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:04.880450 7598 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:44:04.881201 7613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.881486 7598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:04.881557 7598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "2c15d702d9c842e4b9599c0be7c7c6db"
format_stamp: "Formatted at 2026-09-26 20:44:03 on dist-test-slave-4hd3"
server_key: "39ee4ff32c86e9e82c0e328222287f3f"
server_key_iv: "f1ef4b399de612279058f44629c718a3"
server_key_version: "encryptionkey@0"
I20260926 20:44:04.881865 7598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:04.905122 7598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:04.907907 7598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:04.908061 7598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:04.908296 7598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:04.908641 7598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:04.908695 7598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.908730 7598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:04.908763 7598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:04.914788 7598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:45817
I20260926 20:44:04.914829 7726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:45817 every 8 connection(s)
I20260926 20:44:04.915155 7598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:44:04.919836 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 7598
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.124@KRBTEST.COM
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
Sep 26 20:44:04 dist-test-slave-4hd3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
I20260926 20:44:04.927790 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.915900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:51965 (local address 127.0.98.124:40885)
0926 20:44:04.915941 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.915943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.916346 (+ 403us) server_negotiation.cc:407] Connection header received
0926 20:44:04.917659 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.917660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.917673 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.917728 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.918517 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.919009 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.920151 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.920350 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.920427 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.924200 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.924202 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.924202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.924209 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.926120 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.926667 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.926669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.926669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.926729 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.927005 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.927007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.927007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.927140 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.927184 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.927564 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.927644 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:44:04.928041 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.915847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:49067 (local address 127.0.98.125:38537)
0926 20:44:04.915914 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.915916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.916379 (+ 463us) server_negotiation.cc:407] Connection header received
0926 20:44:04.917675 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.917676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.917687 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.917720 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.918538 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.919278 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.922342 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.922668 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.922708 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.924712 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.924713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.924714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.924720 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.926650 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.927145 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.927146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.927147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.927211 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.927424 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.927425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.927425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.927561 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.927604 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.927842 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.927917 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:44:04.928153 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.916354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.67:48803)
0926 20:44:04.916495 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.916522 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.917574 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.917844 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.917845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.918293 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.918424 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.918430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.919374 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.919374 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.920307 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.920310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.920758 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.920759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.920832 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.921296 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.921301 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.923696 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.926742 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.926743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.926743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.927056 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.927269 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.927269 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.927269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.927323 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.927593 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.927593 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.927642 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.927740 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.927929 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":41,"thread_start_us":68,"threads_started":1}
I20260926 20:44:04.928352 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.915876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:38537 (local address 127.0.98.67:49067)
0926 20:44:04.916218 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.916271 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.917574 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.917820 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.917820 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.918291 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.918424 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.918430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.919374 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.919374 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.922238 (+ 2864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.922241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.922783 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.922784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.922841 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.923252 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.923256 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.924621 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.926731 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.926732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.926732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.927057 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.927289 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.927290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.927290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.927352 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.927653 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.927654 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.927703 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.928119 (+ 416us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.928227 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":49,"thread_start_us":239,"threads_started":1}
I20260926 20:44:04.928618 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.916542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:48803 (local address 127.0.98.126:41639)
0926 20:44:04.916589 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.916591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.916599 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:44:04.917674 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.917675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.917687 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.917720 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.918534 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.919283 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.920398 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.920674 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.920747 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.923822 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.923824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.923824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.923832 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.926662 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.927144 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.927145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.927146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.927200 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.927387 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.927388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.927388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.927511 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.927568 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.928444 (+ 876us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.928532 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 20:44:04.928961 7728 heartbeater.cc:347] Connected to a master server at 127.0.98.126:41639
I20260926 20:44:04.929054 7728 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:04.929235 7728 heartbeater.cc:511] Master 127.0.98.126:41639 requested a full tablet report, sending...
I20260926 20:44:04.929414 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.915985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40885 (local address 127.0.98.67:51965)
0926 20:44:04.916230 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.916270 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.917555 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.917806 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.917811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.918232 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:04.918424 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.918428 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.919112 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.919113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.920054 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.920057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.920486 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.920487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.920620 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.921166 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:04.921177 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:04.924103 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:04.926213 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.926219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.926221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.926570 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.926801 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:04.926802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:04.926802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:04.926924 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:04.927232 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:04.927238 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:04.927409 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.928592 (+ 1183us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.929127 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":214,"threads_started":1}
I20260926 20:44:04.929687 6491 ts_manager.cc:194] Re-registered known tserver with Master: 2c15d702d9c842e4b9599c0be7c7c6db (127.0.98.67:45817)
I20260926 20:44:04.929919 7727 heartbeater.cc:347] Connected to a master server at 127.0.98.124:40885
I20260926 20:44:04.929960 7727 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:04.930016 7727 heartbeater.cc:511] Master 127.0.98.124:40885 requested a full tablet report, sending...
I20260926 20:44:04.930121 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.921000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.0.98.126:41639)
0926 20:44:04.921129 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.921131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.921802 (+ 671us) server_negotiation.cc:407] Connection header received
0926 20:44:04.921827 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.921828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.921856 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.921930 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.922751 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.923540 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.924687 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.925009 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.925097 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.926682 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.926683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.926683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.926691 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.928820 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.929172 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.929173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.929174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.929232 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.929439 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.929440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.929440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.929564 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.929654 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.929944 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.930031 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260926 20:44:04.930171 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:48803
I20260926 20:44:04.930244 7274 ts_manager.cc:194] Registered new tserver with Master: 2c15d702d9c842e4b9599c0be7c7c6db (127.0.98.67:45817)
I20260926 20:44:04.930365 7729 heartbeater.cc:347] Connected to a master server at 127.0.98.125:38537
I20260926 20:44:04.930403 7729 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:04.930938 7729 heartbeater.cc:511] Master 127.0.98.125:38537 requested a full tablet report, sending...
I20260926 20:44:04.931438 7089 ts_manager.cc:194] Registered new tserver with Master: 2c15d702d9c842e4b9599c0be7c7c6db (127.0.98.67:45817)
I20260926 20:44:04.933629 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.928155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37898 (local address 127.0.98.124:40885)
0926 20:44:04.928188 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.928189 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.929269 (+ 1080us) server_negotiation.cc:407] Connection header received
0926 20:44:04.929368 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.929369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.929391 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.929430 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:04.929613 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.930281 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.932625 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.933175 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.933220 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.933316 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.933538 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.933571 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 20:44:04.933847 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.928021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40885 (local address 127.0.0.1:37898)
0926 20:44:04.929214 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.929224 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.929295 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.929500 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.929501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.929503 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:04.929536 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.929537 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.930664 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.930664 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.932530 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.932532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.933387 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.933388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.933449 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.933451 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.933743 (+ 292us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.933784 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"thread_start_us":54,"threads_started":1}
I20260926 20:44:04.934070 7284 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 20:44:04.931844 6491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 20:44:04.934259 7313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader eca7d68fee6a46ec8f522d2a7203f3ef. Latest consensus state: current_term: 1 leader_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: true } } }
I20260926 20:44:04.934335 7313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [sys.catalog]: This master's current role is: LEARNER
W20260926 20:44:04.934413 6491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:44:04.935247 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: true } } }
I20260926 20:44:04.935704 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:04.937165 7746 raft_consensus.cc:1050] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef: attempting to promote NON_VOTER 68fe425de5074e879f7c0e4c90d6b4bd to VOTER
I20260926 20:44:04.937296 7746 consensus_queue.cc:237] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } }
I20260926 20:44:04.937856 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.921535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37886 (local address 127.0.98.124:40885)
0926 20:44:04.921783 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.921785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.922119 (+ 334us) server_negotiation.cc:407] Connection header received
0926 20:44:04.922139 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.922140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.922159 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.922211 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.923005 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.923736 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.932467 (+ 8731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.932762 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.932804 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.934451 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.934452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.934453 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.934459 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.936427 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.936837 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.936837 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.936838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.936899 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.937128 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.937129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.937129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.937259 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.937646 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.937687 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.937768 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":53,"threads_started":1}
I20260926 20:44:04.937860 7098 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 1 FOLLOWER]: Refusing update from remote peer eca7d68fee6a46ec8f522d2a7203f3ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 20:44:04.937875 7284 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 1 LEARNER]: Refusing update from remote peer eca7d68fee6a46ec8f522d2a7203f3ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 20:44:04.938037 7747 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 20:44:04.938603 7747 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 20:44:04.938566 7746 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 68fe425de5074e879f7c0e4c90d6b4bd (127.0.98.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } } }
I20260926 20:44:04.938771 7099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 68fe425de5074e879f7c0e4c90d6b4bd (127.0.98.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } } }
I20260926 20:44:04.938952 7284 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 68fe425de5074e879f7c0e4c90d6b4bd (127.0.98.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } } }
I20260926 20:44:04.939380 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.921340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33616 (local address 127.0.98.125:38537)
0926 20:44:04.924967 (+ 3627us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.924970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.924979 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:44:04.925105 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.925106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.925119 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.925190 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:04.931435 (+ 6245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.932244 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.933344 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.934110 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.934213 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.936177 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:04.936178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:04.936179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:04.936185 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:04.937977 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.938336 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.938337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.938337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.938417 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:04.938623 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:04.938624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:04.938625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:04.938767 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:04.938868 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.939224 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.939302 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3192,"thread_start_us":52,"threads_started":1}
I20260926 20:44:04.939613 7746 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } } }
I20260926 20:44:04.939688 7746 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:04.939697 7754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } } }
I20260926 20:44:04.939775 7746 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } } }
I20260926 20:44:04.939814 7754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:44:04.939836 7746 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:04.940053 7313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } } }
I20260926 20:44:04.940132 7313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:44:04.941296 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } } }
I20260926 20:44:04.941394 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:04.946902 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.942785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.0.98.65:36235)
0926 20:44:04.942953 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.942958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.942969 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:44:04.943009 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.943011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.943026 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.943112 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:04.943215 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.944326 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.946104 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.946566 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.946654 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.946731 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.946788 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.946824 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260926 20:44:04.946890 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.942550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:36235 (local address 127.0.0.1:55668)
0926 20:44:04.942587 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.942598 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.942635 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.943114 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.943115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.943117 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:04.943159 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.943162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.944411 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.944413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.945507 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.945509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.946654 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.946655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.946710 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.946714 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.946772 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.946818 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 20:44:04.947197 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.942643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:42795 (local address 127.0.0.1:54262)
0926 20:44:04.943042 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.943052 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.943153 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.943763 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.943764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.943766 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:04.943820 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.943821 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.945011 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.945011 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.946027 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.946029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.946924 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.946925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.946991 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.946997 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.947047 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.947099 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":90,"threads_started":1}
I20260926 20:44:04.947304 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.942799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38962 (local address 127.0.98.67:45817)
0926 20:44:04.943272 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.943276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.943284 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:44:04.943328 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.943330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.943356 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.943421 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:04.943592 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.944602 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.945833 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.946549 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.946638 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.946711 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.947175 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.947219 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":98,"threads_started":1}
I20260926 20:44:04.947314 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.942563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:45817 (local address 127.0.0.1:38962)
0926 20:44:04.942766 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:04.942776 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:04.942815 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:04.943443 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:04.943444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:04.943447 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:04.943496 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.943498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.944702 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.944702 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.945756 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:04.945758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.947007 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.947008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:04.947073 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.947076 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:44:04.947144 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 20:44:04.947186 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
I20260926 20:44:04.947553 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:04.942866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54262 (local address 127.0.98.66:42795)
0926 20:44:04.943574 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:04.943577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:04.943585 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:44:04.943614 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:04.943616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:04.943632 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:04.943724 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:04.943887 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.944917 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.946102 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:04.946811 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:04.946920 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:04.947216 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:04.947245 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 20:44:04.947471 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":83,"threads_started":1}
I20260926 20:44:04.948038 7389 tablet_service.cc:1467] Processing CreateTablet for tablet acc4c0ee667d4c00b71ef9048aebb996 (DEFAULT_TABLE table=test-table [id=4eaa2a4fc4e5477d9f86e1b4b5467899]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:04.948419 7389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acc4c0ee667d4c00b71ef9048aebb996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:04.948410 7661 tablet_service.cc:1467] Processing CreateTablet for tablet acc4c0ee667d4c00b71ef9048aebb996 (DEFAULT_TABLE table=test-table [id=4eaa2a4fc4e5477d9f86e1b4b5467899]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:04.948673 7661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acc4c0ee667d4c00b71ef9048aebb996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:04.951413 7761 tablet_bootstrap.cc:492] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42: Bootstrap starting.
I20260926 20:44:04.951635 7762 tablet_bootstrap.cc:492] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db: Bootstrap starting.
I20260926 20:44:04.951879 7525 tablet_service.cc:1467] Processing CreateTablet for tablet acc4c0ee667d4c00b71ef9048aebb996 (DEFAULT_TABLE table=test-table [id=4eaa2a4fc4e5477d9f86e1b4b5467899]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:04.952602 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acc4c0ee667d4c00b71ef9048aebb996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:04.952666 7762 tablet_bootstrap.cc:654] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:04.952780 7761 tablet_bootstrap.cc:654] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:04.952934 7762 log.cc:824] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db: Log is configured to *not* fsync() on all Append() calls
I20260926 20:44:04.953184 7761 log.cc:824] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42: Log is configured to *not* fsync() on all Append() calls
I20260926 20:44:04.953640 7762 tablet_bootstrap.cc:492] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db: No bootstrap required, opened a new log
I20260926 20:44:04.953727 7762 ts_tablet_manager.cc:1397] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:44:04.955518 7765 tablet_bootstrap.cc:492] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed: Bootstrap starting.
I20260926 20:44:04.956036 7762 raft_consensus.cc:348] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } } peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } }
I20260926 20:44:04.956254 7765 tablet_bootstrap.cc:654] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:04.956254 7762 raft_consensus.cc:374] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:04.956295 7762 raft_consensus.cc:729] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c15d702d9c842e4b9599c0be7c7c6db, State: Initialized, Role: FOLLOWER
I20260926 20:44:04.956342 7761 tablet_bootstrap.cc:492] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42: No bootstrap required, opened a new log
I20260926 20:44:04.956400 7761 ts_tablet_manager.cc:1397] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 20:44:04.956470 7762 consensus_queue.cc:260] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } } peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } }
I20260926 20:44:04.956599 7765 log.cc:824] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed: Log is configured to *not* fsync() on all Append() calls
I20260926 20:44:04.956792 7728 heartbeater.cc:503] Master 127.0.98.126:41639 was elected leader, sending a full tablet report...
I20260926 20:44:04.956755 7762 ts_tablet_manager.cc:1428] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:44:04.957911 7765 tablet_bootstrap.cc:492] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed: No bootstrap required, opened a new log
I20260926 20:44:04.957967 7765 ts_tablet_manager.cc:1397] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 20:44:04.959544 7761 raft_consensus.cc:348] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } } peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } }
I20260926 20:44:04.959702 7761 raft_consensus.cc:374] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:04.959769 7761 raft_consensus.cc:729] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789f62973fe7453a87bc5bd18132ba42, State: Initialized, Role: FOLLOWER
I20260926 20:44:04.959903 7761 consensus_queue.cc:260] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } } peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } }
I20260926 20:44:04.960187 7761 ts_tablet_manager.cc:1428] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260926 20:44:04.960207 7456 heartbeater.cc:503] Master 127.0.98.126:41639 was elected leader, sending a full tablet report...
I20260926 20:44:04.960965 7765 raft_consensus.cc:348] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } } peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } }
I20260926 20:44:04.961135 7765 raft_consensus.cc:374] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:04.961179 7765 raft_consensus.cc:729] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7236e5c160a440aa70f7e343d1623ed, State: Initialized, Role: FOLLOWER
I20260926 20:44:04.961294 7765 consensus_queue.cc:260] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } } peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } }
I20260926 20:44:04.961503 7765 ts_tablet_manager.cc:1428] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260926 20:44:04.961614 7592 heartbeater.cc:503] Master 127.0.98.126:41639 was elected leader, sending a full tablet report...
I20260926 20:44:05.003656 7769 raft_consensus.cc:482] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:44:05.003870 7769 raft_consensus.cc:504] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } } peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } }
I20260926 20:44:05.004321 7769 leader_election.cc:288] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 789f62973fe7453a87bc5bd18132ba42 (127.0.98.65:36235), 2c15d702d9c842e4b9599c0be7c7c6db (127.0.98.67:45817)
I20260926 20:44:05.007017 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:05.004521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:37287 (local address 127.0.98.65:36235)
0926 20:44:05.004563 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:05.004564 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:05.004574 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:44:05.004657 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:05.004658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:05.004670 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:05.004744 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:05.004923 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:05.005392 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:05.006291 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:05.006675 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:05.006754 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:05.006765 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:05.006916 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 20:44:05.006951 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:44:05.007019 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:05.004471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:36235 (local address 127.0.98.66:37287)
0926 20:44:05.004525 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:05.004537 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:05.004581 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:05.004747 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:05.004748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:05.004750 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:05.004838 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:05.004840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:05.005470 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:05.005471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:05.006201 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:05.006203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:05.006763 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:05.006763 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:05.006834 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:05.006837 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:44:05.006909 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 20:44:05.006949 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":4}
I20260926 20:44:05.007094 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:05.004573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:43629 (local address 127.0.98.67:45817)
0926 20:44:05.004614 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:05.004616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:05.004624 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:44:05.004732 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:05.004733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:05.004745 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:05.004791 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:05.004960 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:05.005421 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:05.006303 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:05.006732 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:05.006813 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:05.006830 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:05.006982 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 20:44:05.007022 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:44:05.007125 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:05.004534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:45817 (local address 127.0.98.66:43629)
0926 20:44:05.004586 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:05.004596 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:05.004637 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:05.004803 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:05.004804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:05.004805 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:05.004872 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:05.004876 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:05.005503 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:05.005504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:05.006200 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:05.006202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:05.006814 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:05.006815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:05.006896 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:05.006898 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:44:05.006974 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:44:05.007015 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":3}
I20260926 20:44:05.007449 7409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc4c0ee667d4c00b71ef9048aebb996" candidate_uuid: "a7236e5c160a440aa70f7e343d1623ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789f62973fe7453a87bc5bd18132ba42" is_pre_election: true
I20260926 20:44:05.007493 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc4c0ee667d4c00b71ef9048aebb996" candidate_uuid: "a7236e5c160a440aa70f7e343d1623ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" is_pre_election: true
I20260926 20:44:05.007611 7409 raft_consensus.cc:2454] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7236e5c160a440aa70f7e343d1623ed in term 0.
I20260926 20:44:05.007625 7681 raft_consensus.cc:2454] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7236e5c160a440aa70f7e343d1623ed in term 0.
I20260926 20:44:05.007859 7481 leader_election.cc:302] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 789f62973fe7453a87bc5bd18132ba42, a7236e5c160a440aa70f7e343d1623ed; no voters:
I20260926 20:44:05.008037 7769 raft_consensus.cc:2781] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:44:05.008129 7769 raft_consensus.cc:482] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:44:05.008162 7769 raft_consensus.cc:3037] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:05.008889 7769 raft_consensus.cc:504] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } } peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } }
I20260926 20:44:05.009063 7769 leader_election.cc:288] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [CANDIDATE]: Term 1 election: Requested vote from peers 789f62973fe7453a87bc5bd18132ba42 (127.0.98.65:36235), 2c15d702d9c842e4b9599c0be7c7c6db (127.0.98.67:45817)
I20260926 20:44:05.009207 7409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc4c0ee667d4c00b71ef9048aebb996" candidate_uuid: "a7236e5c160a440aa70f7e343d1623ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789f62973fe7453a87bc5bd18132ba42"
I20260926 20:44:05.009219 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc4c0ee667d4c00b71ef9048aebb996" candidate_uuid: "a7236e5c160a440aa70f7e343d1623ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c15d702d9c842e4b9599c0be7c7c6db"
I20260926 20:44:05.009284 7409 raft_consensus.cc:3037] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:05.009341 7681 raft_consensus.cc:3037] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:05.009889 7409 raft_consensus.cc:2454] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7236e5c160a440aa70f7e343d1623ed in term 1.
I20260926 20:44:05.010046 7481 leader_election.cc:302] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 789f62973fe7453a87bc5bd18132ba42, a7236e5c160a440aa70f7e343d1623ed; no voters:
I20260926 20:44:05.010156 7681 raft_consensus.cc:2454] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7236e5c160a440aa70f7e343d1623ed in term 1.
I20260926 20:44:05.010160 7769 raft_consensus.cc:2781] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:44:05.010367 7769 raft_consensus.cc:686] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [term 1 LEADER]: Becoming Leader. State: Replica: a7236e5c160a440aa70f7e343d1623ed, State: Running, Role: LEADER
I20260926 20:44:05.010464 7769 consensus_queue.cc:237] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } } peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } }
I20260926 20:44:05.011365 6491 catalog_manager.cc:5495] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed reported cstate change: term changed from 0 to 1, leader changed from <none> to a7236e5c160a440aa70f7e343d1623ed (127.0.98.66). New cstate: current_term: 1 leader_uuid: "a7236e5c160a440aa70f7e343d1623ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } health_report { overall_health: HEALTHY } } }
W20260926 20:44:05.019541 7594 tablet.cc:2369] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:44:05.020102 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:05.017766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54278 (local address 127.0.98.66:42795)
0926 20:44:05.017820 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:05.017822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:05.017831 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:44:05.017902 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:05.017902 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:05.017919 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:05.017982 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:44:05.018115 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:05.018564 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:05.019233 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:05.019488 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:05.019542 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:05.019757 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:44:05.019883 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:44:05.019946 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:05.020010 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 20:44:05.020028 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 20:44:05.021809 7409 raft_consensus.cc:1261] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42 [term 1 FOLLOWER]: Refusing update from remote peer a7236e5c160a440aa70f7e343d1623ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:44:05.021842 7681 raft_consensus.cc:1261] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db [term 1 FOLLOWER]: Refusing update from remote peer a7236e5c160a440aa70f7e343d1623ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:44:05.022130 7770 consensus_queue.cc:1035] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:44:05.022255 7769 consensus_queue.cc:1035] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:44:05.025044 7774 mvcc.cc:205] Tried to move back new op lower bound from 7333705502806953984 to 7333705502763458560. Current Snapshot: MvccSnapshot[applied={T|T < 7333705502806953984}]
I20260926 20:44:05.025661 7775 mvcc.cc:205] Tried to move back new op lower bound from 7333705502806953984 to 7333705502763458560. Current Snapshot: MvccSnapshot[applied={T|T < 7333705502806953984}]
I20260926 20:44:05.027223 6491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:44:05.027308 6491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:44:05.028716 6491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eca7d68fee6a46ec8f522d2a7203f3ef: Sending DeleteTablet for 3 replicas of tablet acc4c0ee667d4c00b71ef9048aebb996
I20260926 20:44:05.028996 7389 tablet_service.cc:1514] Processing DeleteTablet for tablet acc4c0ee667d4c00b71ef9048aebb996 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:05 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:55668
I20260926 20:44:05.029023 7661 tablet_service.cc:1514] Processing DeleteTablet for tablet acc4c0ee667d4c00b71ef9048aebb996 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:05 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:38962
I20260926 20:44:05.029022 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet acc4c0ee667d4c00b71ef9048aebb996 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:05 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:54262
I20260926 20:44:05.029276 7783 tablet_replica.cc:330] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db: stopping tablet replica
I20260926 20:44:05.029384 7784 tablet_replica.cc:330] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed: stopping tablet replica
I20260926 20:44:05.029382 7783 raft_consensus.cc:2229] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:44:05.029451 7784 raft_consensus.cc:2229] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:44:05.029477 7783 raft_consensus.cc:2258] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:05.029618 7782 tablet_replica.cc:330] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42: stopping tablet replica
I20260926 20:44:05.029702 7782 raft_consensus.cc:2229] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:44:05.030015 7783 ts_tablet_manager.cc:1905] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:05.030055 7782 raft_consensus.cc:2258] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:05.030099 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 6459
I20260926 20:44:05.030758 7782 ts_tablet_manager.cc:1905] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:05.029525 7784 raft_consensus.cc:2258] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:05.031939 7783 ts_tablet_manager.cc:1918] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:44:05.032045 7783 log.cc:1196] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/acc4c0ee667d4c00b71ef9048aebb996
I20260926 20:44:05.032222 7784 ts_tablet_manager.cc:1905] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:05.032401 7783 ts_tablet_manager.cc:1939] T acc4c0ee667d4c00b71ef9048aebb996 P 2c15d702d9c842e4b9599c0be7c7c6db: Deleting consensus metadata
I20260926 20:44:05.032400 7782 ts_tablet_manager.cc:1918] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:44:05.032466 7782 log.cc:1196] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-0/wal/wals/acc4c0ee667d4c00b71ef9048aebb996
I20260926 20:44:05.032724 7782 ts_tablet_manager.cc:1939] T acc4c0ee667d4c00b71ef9048aebb996 P 789f62973fe7453a87bc5bd18132ba42: Deleting consensus metadata
I20260926 20:44:05.033751 7784 ts_tablet_manager.cc:1918] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:44:05.033806 7784 log.cc:1196] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/acc4c0ee667d4c00b71ef9048aebb996
I20260926 20:44:05.034044 7784 ts_tablet_manager.cc:1939] T acc4c0ee667d4c00b71ef9048aebb996 P a7236e5c160a440aa70f7e343d1623ed: Deleting consensus metadata
W20260926 20:44:05.036423 7479 connection.cc:535] server connection from 127.0.0.1:54262 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 20:44:05.036458 7617 connection.cc:535] server connection from 127.0.0.1:38962 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 20:44:05.036608 7343 connection.cc:535] server connection from 127.0.0.1:55668 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:44:05.045920 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:05.038277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.0.98.124:40885)
0926 20:44:05.038317 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:05.038319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:05.038510 (+ 191us) server_negotiation.cc:407] Connection header received
0926 20:44:05.038621 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:05.038622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:05.038635 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:05.038684 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:05.039206 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:05.040061 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:05.040822 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:05.041054 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:05.041136 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:05.042773 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:05.042774 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:05.042775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:05.042782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:05.044673 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:05.045072 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:05.045073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:05.045073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:05.045153 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:05.045368 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:05.045369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:05.045370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:05.045496 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:05.045607 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:05.045711 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 20:44:05.045795 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:44:05.046084 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:05.038475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.0.98.125:38537)
0926 20:44:05.038512 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:05.038514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:05.038522 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:44:05.038620 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:05.038621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:05.038636 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:05.038665 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:05.039386 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:05.040151 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:05.041255 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:05.041561 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:05.041652 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:05.043087 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:05.043089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:05.043089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:05.043096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:05.044844 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:05.045198 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:05.045200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:05.045200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:05.045257 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:05.045477 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:05.045478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:05.045479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:05.045615 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:05.045686 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:05.045903 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 20:44:05.045988 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:44:06.025831 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:06.025445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.66:35313)
0926 20:44:06.025642 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:06.025764 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":60,"threads_started":1}
W20260926 20:44:06.026120 7592 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:41639 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
I20260926 20:44:06.026831 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:06.026592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.67:37441)
0926 20:44:06.026707 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:06.026775 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":54,"threads_started":1}
I20260926 20:44:06.026860 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:06.026647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.65:54385)
0926 20:44:06.026762 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:06.026819 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":52,"threads_started":1}
W20260926 20:44:06.027015 7728 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:41639 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
W20260926 20:44:06.027117 7456 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:41639 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
I20260926 20:44:06.529580 7796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eca7d68fee6a46ec8f522d2a7203f3ef)
I20260926 20:44:06.529577 7795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eca7d68fee6a46ec8f522d2a7203f3ef)
I20260926 20:44:06.529706 7796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } }
I20260926 20:44:06.529739 7795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } }
I20260926 20:44:06.530019 7796 leader_election.cc:288] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eca7d68fee6a46ec8f522d2a7203f3ef (127.0.98.126:41639), 2ca257ec00324f9ba9742990fd00c469 (127.0.98.125:38537)
I20260926 20:44:06.530027 7795 leader_election.cc:288] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eca7d68fee6a46ec8f522d2a7203f3ef (127.0.98.126:41639), 68fe425de5074e879f7c0e4c90d6b4bd (127.0.98.124:40885)
I20260926 20:44:06.530349 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:06.530119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:49768)
0926 20:44:06.530235 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:06.530294 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":43,"threads_started":1}
W20260926 20:44:06.530481 7073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
I20260926 20:44:06.530658 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:06.530134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:49772)
0926 20:44:06.530566 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:06.530615 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":52,"threads_started":1}
W20260926 20:44:06.530802 7258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
I20260926 20:44:06.531029 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:06.530902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:49788)
0926 20:44:06.530978 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:06.531005 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 20:44:06.531096 7073 leader_election.cc:334] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer eca7d68fee6a46ec8f522d2a7203f3ef (127.0.98.126:41639): Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
I20260926 20:44:06.531459 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:06.531350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:49796)
0926 20:44:06.531405 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:06.531435 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 20:44:06.531591 7258 leader_election.cc:334] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer eca7d68fee6a46ec8f522d2a7203f3ef (127.0.98.126:41639): Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
I20260926 20:44:06.533422 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:06.530207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40885 (local address 127.0.0.1:37904)
0926 20:44:06.530414 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:06.530428 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:06.530480 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:06.530706 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:06.530707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:06.530710 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:06.530773 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:06.530775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:06.531775 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:06.531776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:06.532570 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:06.532572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:06.533188 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:06.533189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:06.533251 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:06.533254 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:44:06.533318 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 20:44:06.533357 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":48,"threads_started":1}
I20260926 20:44:06.533432 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:06.530339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37904 (local address 127.0.98.124:40885)
0926 20:44:06.530560 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:06.530562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:06.530569 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:44:06.530595 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:06.530596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:06.530608 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:06.530836 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:06.530873 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:06.531644 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:06.532676 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:06.533099 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:06.533176 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:06.533187 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:06.533322 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 20:44:06.533366 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":46,"threads_started":1}
I20260926 20:44:06.533538 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:06.530414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.0.98.125:38537)
0926 20:44:06.530511 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:06.530513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:06.530843 (+ 330us) server_negotiation.cc:407] Connection header received
0926 20:44:06.530868 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:06.530869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:06.530879 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:06.530947 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:06.531135 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:06.531859 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:06.532811 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:06.533192 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:06.533285 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:06.533295 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:06.533443 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 20:44:06.533480 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":39,"threads_started":1}
I20260926 20:44:06.533789 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:06.530276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:38537 (local address 127.0.0.1:33628)
0926 20:44:06.530586 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:06.530595 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:06.530645 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:06.531004 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:06.531004 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:06.531007 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:06.531055 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:06.531058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:06.531944 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:06.531945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:06.532743 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:06.532746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:06.533290 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:06.533291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:06.533354 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:06.533357 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:44:06.533436 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:44:06.533479 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":45,"threads_started":1}
I20260926 20:44:06.533834 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ca257ec00324f9ba9742990fd00c469" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" is_pre_election: true
I20260926 20:44:06.533928 7284 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca257ec00324f9ba9742990fd00c469 in term 1.
I20260926 20:44:06.534018 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2ca257ec00324f9ba9742990fd00c469" is_pre_election: true
I20260926 20:44:06.534060 7074 leader_election.cc:302] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ca257ec00324f9ba9742990fd00c469, 68fe425de5074e879f7c0e4c90d6b4bd; no voters: eca7d68fee6a46ec8f522d2a7203f3ef
I20260926 20:44:06.534119 7099 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68fe425de5074e879f7c0e4c90d6b4bd in term 1.
I20260926 20:44:06.534178 7795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 20:44:06.534225 7795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eca7d68fee6a46ec8f522d2a7203f3ef)
I20260926 20:44:06.534246 7795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:44:06.534255 7258 leader_election.cc:302] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ca257ec00324f9ba9742990fd00c469, 68fe425de5074e879f7c0e4c90d6b4bd; no voters: eca7d68fee6a46ec8f522d2a7203f3ef
I20260926 20:44:06.534345 7796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 20:44:06.534406 7796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 1 FOLLOWER]: Starting leader election (detected failure of leader eca7d68fee6a46ec8f522d2a7203f3ef)
I20260926 20:44:06.534443 7796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:44:06.534976 7795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } }
I20260926 20:44:06.535121 7795 leader_election.cc:288] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [CANDIDATE]: Term 2 election: Requested vote from peers eca7d68fee6a46ec8f522d2a7203f3ef (127.0.98.126:41639), 68fe425de5074e879f7c0e4c90d6b4bd (127.0.98.124:40885)
I20260926 20:44:06.535077 7796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } }
I20260926 20:44:06.535213 7796 leader_election.cc:288] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [CANDIDATE]: Term 2 election: Requested vote from peers eca7d68fee6a46ec8f522d2a7203f3ef (127.0.98.126:41639), 2ca257ec00324f9ba9742990fd00c469 (127.0.98.125:38537)
I20260926 20:44:06.535275 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:06.535189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:49806)
0926 20:44:06.535225 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:06.535250 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 20:44:06.535295 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ca257ec00324f9ba9742990fd00c469" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "68fe425de5074e879f7c0e4c90d6b4bd"
I20260926 20:44:06.535394 7284 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2ca257ec00324f9ba9742990fd00c469 in current term 2: Already voted for candidate 68fe425de5074e879f7c0e4c90d6b4bd in this term.
I20260926 20:44:06.535446 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2ca257ec00324f9ba9742990fd00c469"
I20260926 20:44:06.535521 7099 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 68fe425de5074e879f7c0e4c90d6b4bd in current term 2: Already voted for candidate 2ca257ec00324f9ba9742990fd00c469 in this term.
I20260926 20:44:06.535624 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:06.535523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:49826)
0926 20:44:06.535574 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:06.535600 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 20:44:06.535674 7073 leader_election.cc:334] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer eca7d68fee6a46ec8f522d2a7203f3ef (127.0.98.126:41639): Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
I20260926 20:44:06.535699 7073 leader_election.cc:302] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ca257ec00324f9ba9742990fd00c469; no voters: 68fe425de5074e879f7c0e4c90d6b4bd, eca7d68fee6a46ec8f522d2a7203f3ef
I20260926 20:44:06.535776 7795 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 20:44:06.535843 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:06.535334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:49816)
0926 20:44:06.535775 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:06.535817 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 20:44:06.536105 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:06.536051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:49838)
0926 20:44:06.536073 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:06.536093 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260926 20:44:06.536187 7258 leader_election.cc:334] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer eca7d68fee6a46ec8f522d2a7203f3ef (127.0.98.126:41639): Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
I20260926 20:44:06.536237 7258 leader_election.cc:302] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68fe425de5074e879f7c0e4c90d6b4bd; no voters: 2ca257ec00324f9ba9742990fd00c469, eca7d68fee6a46ec8f522d2a7203f3ef
I20260926 20:44:06.536326 7796 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 20:44:07.026871 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:07.026529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.66:34583)
0926 20:44:07.026760 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:07.026808 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260926 20:44:07.027504 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:07.027299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.67:41479)
0926 20:44:07.027425 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:07.027458 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260926 20:44:07.027614 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:07.027400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.65:38567)
0926 20:44:07.027521 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:07.027556 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
I20260926 20:44:08.027937 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.027401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.66:57823)
0926 20:44:08.027807 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.027856 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":65,"threads_started":1}
I20260926 20:44:08.028177 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.027862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.67:51183)
0926 20:44:08.028084 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.028122 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20260926 20:44:08.028308 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.028071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.65:55193)
0926 20:44:08.028227 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.028263 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":57,"threads_started":1}
I20260926 20:44:08.489259 7811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:44:08.489385 7811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } }
I20260926 20:44:08.489602 7811 leader_election.cc:288] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eca7d68fee6a46ec8f522d2a7203f3ef (127.0.98.126:41639), 2ca257ec00324f9ba9742990fd00c469 (127.0.98.125:38537)
I20260926 20:44:08.490007 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.489784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:50106)
0926 20:44:08.489926 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.489969 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":49,"threads_started":1}
I20260926 20:44:08.490061 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2ca257ec00324f9ba9742990fd00c469" is_pre_election: true
I20260926 20:44:08.490206 7099 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68fe425de5074e879f7c0e4c90d6b4bd in term 2.
I20260926 20:44:08.490448 7258 leader_election.cc:302] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca257ec00324f9ba9742990fd00c469, 68fe425de5074e879f7c0e4c90d6b4bd; no voters:
I20260926 20:44:08.490494 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.490391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:50118)
0926 20:44:08.490444 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.490473 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 20:44:08.490540 7258 leader_election.cc:334] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer eca7d68fee6a46ec8f522d2a7203f3ef (127.0.98.126:41639): Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
I20260926 20:44:08.490568 7811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 20:44:08.490603 7811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:44:08.490635 7811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:44:08.491354 7811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } }
I20260926 20:44:08.491487 7811 leader_election.cc:288] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [CANDIDATE]: Term 3 election: Requested vote from peers eca7d68fee6a46ec8f522d2a7203f3ef (127.0.98.126:41639), 2ca257ec00324f9ba9742990fd00c469 (127.0.98.125:38537)
I20260926 20:44:08.491689 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.491603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:50130)
0926 20:44:08.491648 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.491674 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 20:44:08.491796 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2ca257ec00324f9ba9742990fd00c469"
I20260926 20:44:08.491881 7099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:44:08.492019 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.491942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:50140)
0926 20:44:08.491974 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.492000 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260926 20:44:08.492097 7258 leader_election.cc:334] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer eca7d68fee6a46ec8f522d2a7203f3ef (127.0.98.126:41639): Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
I20260926 20:44:08.492576 7099 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fe425de5074e879f7c0e4c90d6b4bd in term 3.
I20260926 20:44:08.492777 7258 leader_election.cc:302] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ca257ec00324f9ba9742990fd00c469, 68fe425de5074e879f7c0e4c90d6b4bd; no voters: eca7d68fee6a46ec8f522d2a7203f3ef
I20260926 20:44:08.492868 7811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 3 FOLLOWER]: Leader election won for term 3
I20260926 20:44:08.492970 7811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [term 3 LEADER]: Becoming Leader. State: Replica: 68fe425de5074e879f7c0e4c90d6b4bd, State: Running, Role: LEADER
I20260926 20:44:08.493026 7811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } }
I20260926 20:44:08.493391 7814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68fe425de5074e879f7c0e4c90d6b4bd. Latest consensus state: current_term: 3 leader_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } } }
I20260926 20:44:08.493465 7814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:08.493635 7816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:44:08.494053 7816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:44:08.494153 7816 catalog_manager.cc:1259] Loaded cluster ID: 6c31914beaf84aacb1952b002eda74f0
I20260926 20:44:08.494194 7816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:44:08.494381 7816 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:44:08.494477 7816 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: Loaded TSK: 0
I20260926 20:44:08.494620 7816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:44:08.541725 7272 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:44:08.543862 7272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:44:08.544907 7099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [term 3 FOLLOWER]: Refusing update from remote peer 68fe425de5074e879f7c0e4c90d6b4bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 20:44:08.544962 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.544844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:50160)
0926 20:44:08.544894 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.544934 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:44:08.545169 7811 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 20:44:08.545331 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.545214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.0.1:50170)
0926 20:44:08.545258 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.545283 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 20:44:08.545424 7258 consensus_peers.cc:487] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd -> Peer eca7d68fee6a46ec8f522d2a7203f3ef (127.0.98.126:41639): Couldn't send request to peer eca7d68fee6a46ec8f522d2a7203f3ef. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 20:44:08.545861 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68fe425de5074e879f7c0e4c90d6b4bd. Latest consensus state: current_term: 3 leader_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } } }
I20260926 20:44:08.545976 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:44:08.546443 7811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } } }
I20260926 20:44:08.546542 7811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:08.546543 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eca7d68fee6a46ec8f522d2a7203f3ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41639 } } peers { permanent_uuid: "2ca257ec00324f9ba9742990fd00c469" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 38537 } attrs { promote: false } } peers { permanent_uuid: "68fe425de5074e879f7c0e4c90d6b4bd" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40885 } attrs { promote: false } } }
I20260926 20:44:08.546639 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca257ec00324f9ba9742990fd00c469 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:44:08.546746 7819 mvcc.cc:205] Tried to move back new op lower bound from 7333705517238001664 to 7333705517027950592. Current Snapshot: MvccSnapshot[applied={T|T < 7333705517238001664 or (T in {7333705517238001664})}]
I20260926 20:44:08.551501 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.548597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54294 (local address 127.0.98.66:42795)
0926 20:44:08.548709 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:08.548711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:08.548719 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:44:08.548747 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:08.548748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:08.548759 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:08.548823 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:08.549068 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:08.549813 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:08.550744 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:08.551195 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:08.551264 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:08.551277 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:08.551400 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 20:44:08.551433 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":66,"threads_started":1}
I20260926 20:44:08.551508 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.548416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:42795 (local address 127.0.0.1:54294)
0926 20:44:08.548605 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.548615 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:08.548666 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:08.548892 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:08.548893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:08.548894 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:08.548984 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:08.548986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:08.549903 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:08.549904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:08.550648 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:08.550650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:08.551271 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:08.551271 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:08.551327 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:08.551332 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:44:08.551393 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 20:44:08.551439 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":44,"thread_start_us":89,"threads_started":1}
I20260926 20:44:08.551815 7525 tablet_service.cc:1467] Processing CreateTablet for tablet e1758261341e4c9f8badabc05371dbf6 (DEFAULT_TABLE table=test-table [id=66ac27aad5b3453c94b7998615db3841]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:08.551954 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1758261341e4c9f8badabc05371dbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:08.552038 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.548448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:36235 (local address 127.0.0.1:55684)
0926 20:44:08.548628 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.548637 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:08.548675 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:08.548961 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:08.548963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:08.548965 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:08.548998 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:08.549000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:08.549899 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:08.549900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:08.551009 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:08.551011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:08.551814 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:08.551815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:08.551874 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:08.551877 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:44:08.551921 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 20:44:08.551961 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":27,"thread_start_us":63,"threads_started":1}
I20260926 20:44:08.552246 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.548373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:45817 (local address 127.0.0.1:38968)
0926 20:44:08.548413 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.548423 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:08.548471 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:08.548892 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:08.548893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:08.548894 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:08.548973 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:08.548975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:08.549693 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:08.549694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:08.550833 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:08.550835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:08.552056 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:08.552057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:08.552120 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:08.552125 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:44:08.552156 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 20:44:08.552193 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":9}
I20260926 20:44:08.552320 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.548717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55684 (local address 127.0.98.65:36235)
0926 20:44:08.548822 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:08.548825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:08.548834 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:44:08.548866 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:08.548867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:08.548877 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:08.549061 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:08.549097 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:08.549812 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:08.551090 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:08.551692 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:08.551774 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:08.551786 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:08.552197 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 20:44:08.552243 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":69,"threads_started":1}
I20260926 20:44:08.552522 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.548445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.0.98.67:45817)
0926 20:44:08.548768 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:08.548770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:08.548778 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:44:08.548806 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:08.548807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:08.548816 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:08.548875 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:08.549032 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:08.549598 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:08.551162 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:08.551958 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:08.552391 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:08.552405 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:08.552424 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 20:44:08.552460 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":54,"threads_started":1}
I20260926 20:44:08.552737 7661 tablet_service.cc:1467] Processing CreateTablet for tablet e1758261341e4c9f8badabc05371dbf6 (DEFAULT_TABLE table=test-table [id=66ac27aad5b3453c94b7998615db3841]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:08.552865 7661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1758261341e4c9f8badabc05371dbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:08.553483 7828 tablet_bootstrap.cc:492] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed: Bootstrap starting.
I20260926 20:44:08.553526 7389 tablet_service.cc:1467] Processing CreateTablet for tablet e1758261341e4c9f8badabc05371dbf6 (DEFAULT_TABLE table=test-table [id=66ac27aad5b3453c94b7998615db3841]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:08.553660 7389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1758261341e4c9f8badabc05371dbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:08.553874 7828 tablet_bootstrap.cc:654] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:08.554210 7830 tablet_bootstrap.cc:492] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db: Bootstrap starting.
I20260926 20:44:08.554491 7828 tablet_bootstrap.cc:492] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed: No bootstrap required, opened a new log
I20260926 20:44:08.554543 7828 ts_tablet_manager.cc:1397] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:08.554656 7828 raft_consensus.cc:348] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } } peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } }
I20260926 20:44:08.554725 7828 raft_consensus.cc:374] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:08.554739 7828 raft_consensus.cc:729] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7236e5c160a440aa70f7e343d1623ed, State: Initialized, Role: FOLLOWER
I20260926 20:44:08.554795 7828 consensus_queue.cc:260] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } } peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } }
I20260926 20:44:08.555029 7832 tablet_bootstrap.cc:492] T e1758261341e4c9f8badabc05371dbf6 P 789f62973fe7453a87bc5bd18132ba42: Bootstrap starting.
I20260926 20:44:08.555342 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.555155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.66:38613)
0926 20:44:08.555269 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.555305 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":76,"threads_started":1}
I20260926 20:44:08.555430 7832 tablet_bootstrap.cc:654] T e1758261341e4c9f8badabc05371dbf6 P 789f62973fe7453a87bc5bd18132ba42: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:08.554921 7828 ts_tablet_manager.cc:1428] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:44:08.555631 7591 heartbeater.cc:503] Master 127.0.98.124:40885 was elected leader, sending a full tablet report...
I20260926 20:44:08.555863 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.555762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.66:42325)
0926 20:44:08.555792 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.555830 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260926 20:44:08.556209 7830 tablet_bootstrap.cc:654] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:08.556699 7830 tablet_bootstrap.cc:492] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db: No bootstrap required, opened a new log
I20260926 20:44:08.556741 7832 tablet_bootstrap.cc:492] T e1758261341e4c9f8badabc05371dbf6 P 789f62973fe7453a87bc5bd18132ba42: No bootstrap required, opened a new log
I20260926 20:44:08.556775 7830 ts_tablet_manager.cc:1397] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 20:44:08.556802 7832 ts_tablet_manager.cc:1397] T e1758261341e4c9f8badabc05371dbf6 P 789f62973fe7453a87bc5bd18132ba42: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260926 20:44:08.556900 7830 raft_consensus.cc:348] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } } peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } }
I20260926 20:44:08.556969 7830 raft_consensus.cc:374] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:08.556984 7830 raft_consensus.cc:729] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c15d702d9c842e4b9599c0be7c7c6db, State: Initialized, Role: FOLLOWER
I20260926 20:44:08.556964 7832 raft_consensus.cc:348] T e1758261341e4c9f8badabc05371dbf6 P 789f62973fe7453a87bc5bd18132ba42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } } peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } }
I20260926 20:44:08.557039 7832 raft_consensus.cc:374] T e1758261341e4c9f8badabc05371dbf6 P 789f62973fe7453a87bc5bd18132ba42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:08.557066 7832 raft_consensus.cc:729] T e1758261341e4c9f8badabc05371dbf6 P 789f62973fe7453a87bc5bd18132ba42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789f62973fe7453a87bc5bd18132ba42, State: Initialized, Role: FOLLOWER
I20260926 20:44:08.557051 7830 consensus_queue.cc:260] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } } peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } }
I20260926 20:44:08.557117 7832 consensus_queue.cc:260] T e1758261341e4c9f8badabc05371dbf6 P 789f62973fe7453a87bc5bd18132ba42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } } peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } }
I20260926 20:44:08.557216 7830 ts_tablet_manager.cc:1428] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:44:08.557327 7832 ts_tablet_manager.cc:1428] T e1758261341e4c9f8badabc05371dbf6 P 789f62973fe7453a87bc5bd18132ba42: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 20:44:08.557560 7727 heartbeater.cc:503] Master 127.0.98.124:40885 was elected leader, sending a full tablet report...
I20260926 20:44:08.557662 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.557451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.67:36013)
0926 20:44:08.557576 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.557615 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":80,"threads_started":1}
I20260926 20:44:08.557845 7455 heartbeater.cc:503] Master 127.0.98.124:40885 was elected leader, sending a full tablet report...
I20260926 20:44:08.558018 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.557567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.65:35047)
0926 20:44:08.557941 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.557975 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":70,"threads_started":1}
I20260926 20:44:08.558410 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.558325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.67:41433)
0926 20:44:08.558357 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.558383 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 20:44:08.558442 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.558231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.65:50015)
0926 20:44:08.558394 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.558417 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260926 20:44:08.565052 7831 raft_consensus.cc:482] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:44:08.565112 7831 raft_consensus.cc:504] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } } peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } }
I20260926 20:44:08.565311 7831 leader_election.cc:288] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c15d702d9c842e4b9599c0be7c7c6db (127.0.98.67:45817), 789f62973fe7453a87bc5bd18132ba42 (127.0.98.65:36235)
I20260926 20:44:08.565448 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1758261341e4c9f8badabc05371dbf6" candidate_uuid: "a7236e5c160a440aa70f7e343d1623ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" is_pre_election: true
I20260926 20:44:08.565554 7681 raft_consensus.cc:2454] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7236e5c160a440aa70f7e343d1623ed in term 0.
I20260926 20:44:08.565523 7409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1758261341e4c9f8badabc05371dbf6" candidate_uuid: "a7236e5c160a440aa70f7e343d1623ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789f62973fe7453a87bc5bd18132ba42" is_pre_election: true
I20260926 20:44:08.565626 7409 raft_consensus.cc:2454] T e1758261341e4c9f8badabc05371dbf6 P 789f62973fe7453a87bc5bd18132ba42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7236e5c160a440aa70f7e343d1623ed in term 0.
I20260926 20:44:08.565689 7480 leader_election.cc:302] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c15d702d9c842e4b9599c0be7c7c6db, a7236e5c160a440aa70f7e343d1623ed; no voters:
I20260926 20:44:08.565765 7831 raft_consensus.cc:2781] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:44:08.565810 7831 raft_consensus.cc:482] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:44:08.565861 7831 raft_consensus.cc:3037] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:08.566509 7831 raft_consensus.cc:504] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } } peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } }
I20260926 20:44:08.566641 7831 leader_election.cc:288] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [CANDIDATE]: Term 1 election: Requested vote from peers 2c15d702d9c842e4b9599c0be7c7c6db (127.0.98.67:45817), 789f62973fe7453a87bc5bd18132ba42 (127.0.98.65:36235)
I20260926 20:44:08.566783 7409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1758261341e4c9f8badabc05371dbf6" candidate_uuid: "a7236e5c160a440aa70f7e343d1623ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789f62973fe7453a87bc5bd18132ba42"
I20260926 20:44:08.566859 7409 raft_consensus.cc:3037] T e1758261341e4c9f8badabc05371dbf6 P 789f62973fe7453a87bc5bd18132ba42 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:08.567414 7409 raft_consensus.cc:2454] T e1758261341e4c9f8badabc05371dbf6 P 789f62973fe7453a87bc5bd18132ba42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7236e5c160a440aa70f7e343d1623ed in term 1.
I20260926 20:44:08.567507 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1758261341e4c9f8badabc05371dbf6" candidate_uuid: "a7236e5c160a440aa70f7e343d1623ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c15d702d9c842e4b9599c0be7c7c6db"
I20260926 20:44:08.567600 7681 raft_consensus.cc:3037] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:08.567636 7481 leader_election.cc:302] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 789f62973fe7453a87bc5bd18132ba42, a7236e5c160a440aa70f7e343d1623ed; no voters:
I20260926 20:44:08.567757 7831 raft_consensus.cc:2781] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:44:08.567898 7831 raft_consensus.cc:686] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [term 1 LEADER]: Becoming Leader. State: Replica: a7236e5c160a440aa70f7e343d1623ed, State: Running, Role: LEADER
I20260926 20:44:08.567977 7831 consensus_queue.cc:237] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } } peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } }
I20260926 20:44:08.568199 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.568103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.66:32871)
0926 20:44:08.568146 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.568171 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 20:44:08.568571 7681 raft_consensus.cc:2454] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7236e5c160a440aa70f7e343d1623ed in term 1.
I20260926 20:44:08.568562 7272 catalog_manager.cc:5495] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed reported cstate change: term changed from 0 to 1, leader changed from <none> to a7236e5c160a440aa70f7e343d1623ed (127.0.98.66). New cstate: current_term: 1 leader_uuid: "a7236e5c160a440aa70f7e343d1623ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7236e5c160a440aa70f7e343d1623ed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 } health_report { overall_health: UNKNOWN } } }
I20260926 20:44:08.579578 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.577266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54298 (local address 127.0.98.66:42795)
0926 20:44:08.577326 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:08.577328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:08.577392 (+ 64us) server_negotiation.cc:407] Connection header received
0926 20:44:08.577467 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:08.577467 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:08.577480 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:08.577534 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:44:08.577645 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:08.578114 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:08.578882 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:08.579082 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:08.579156 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:08.579305 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:44:08.579343 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:44:08.579416 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:08.579480 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 20:44:08.579501 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:44:08.580492 7681 raft_consensus.cc:1261] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db [term 1 FOLLOWER]: Refusing update from remote peer a7236e5c160a440aa70f7e343d1623ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:44:08.580492 7409 raft_consensus.cc:1261] T e1758261341e4c9f8badabc05371dbf6 P 789f62973fe7453a87bc5bd18132ba42 [term 1 FOLLOWER]: Refusing update from remote peer a7236e5c160a440aa70f7e343d1623ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:44:08.580763 7831 consensus_queue.cc:1035] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c15d702d9c842e4b9599c0be7c7c6db" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:44:08.580914 7840 consensus_queue.cc:1035] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "789f62973fe7453a87bc5bd18132ba42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 36235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:44:08.581508 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.581384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.67:41963)
0926 20:44:08.581440 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.581481 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 20:44:08.581926 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.581680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.65:55233)
0926 20:44:08.581864 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.581896 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260926 20:44:08.582813 7844 mvcc.cc:205] Tried to move back new op lower bound from 7333705517383868416 to 7333705517334970368. Current Snapshot: MvccSnapshot[applied={T|T < 7333705517383868416}]
I20260926 20:44:08.582944 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.582854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.66:49441)
0926 20:44:08.582892 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.582919 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 20:44:08.583160 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.583064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.65:46273)
0926 20:44:08.583101 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.583130 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 20:44:08.584316 7272 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:44:08.584403 7272 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:44:08.585001 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.584835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41639 (local address 127.0.98.67:52525)
0926 20:44:08.584944 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.584974 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 20:44:08.585747 7272 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68fe425de5074e879f7c0e4c90d6b4bd: Sending DeleteTablet for 3 replicas of tablet e1758261341e4c9f8badabc05371dbf6
I20260926 20:44:08.586016 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet e1758261341e4c9f8badabc05371dbf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:08 UTC) from {username='oryx', principal='oryx/127.0.98.124@KRBTEST.COM'} at 127.0.0.1:54294
I20260926 20:44:08.586061 7389 tablet_service.cc:1514] Processing DeleteTablet for tablet e1758261341e4c9f8badabc05371dbf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:08 UTC) from {username='oryx', principal='oryx/127.0.98.124@KRBTEST.COM'} at 127.0.0.1:55684
I20260926 20:44:08.586146 7661 tablet_service.cc:1514] Processing DeleteTablet for tablet e1758261341e4c9f8badabc05371dbf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:08 UTC) from {username='oryx', principal='oryx/127.0.98.124@KRBTEST.COM'} at 127.0.0.1:38968
I20260926 20:44:08.586236 7854 tablet_replica.cc:330] T e1758261341e4c9f8badabc05371dbf6 P 789f62973fe7453a87bc5bd18132ba42: stopping tablet replica
I20260926 20:44:08.586298 7854 raft_consensus.cc:2229] T e1758261341e4c9f8badabc05371dbf6 P 789f62973fe7453a87bc5bd18132ba42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:44:08.586344 7854 raft_consensus.cc:2258] T e1758261341e4c9f8badabc05371dbf6 P 789f62973fe7453a87bc5bd18132ba42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:08.586443 7853 tablet_replica.cc:330] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed: stopping tablet replica
I20260926 20:44:08.586484 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 7326
I20260926 20:44:08.587234 7853 raft_consensus.cc:2229] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:44:08.587344 7853 raft_consensus.cc:2258] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:08.587699 7853 ts_tablet_manager.cc:1905] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:08.586330 7855 tablet_replica.cc:330] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db: stopping tablet replica
I20260926 20:44:08.588037 7855 raft_consensus.cc:2229] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:44:08.588090 7855 raft_consensus.cc:2258] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:08.588335 7855 ts_tablet_manager.cc:1905] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:08.589681 7855 ts_tablet_manager.cc:1918] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:44:08.589731 7855 log.cc:1196] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/e1758261341e4c9f8badabc05371dbf6
I20260926 20:44:08.589932 7855 ts_tablet_manager.cc:1939] T e1758261341e4c9f8badabc05371dbf6 P 2c15d702d9c842e4b9599c0be7c7c6db: Deleting consensus metadata
I20260926 20:44:08.590058 7853 ts_tablet_manager.cc:1918] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:44:08.590155 7853 log.cc:1196] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/e1758261341e4c9f8badabc05371dbf6
I20260926 20:44:08.590198 7260 catalog_manager.cc:4841] TS 2c15d702d9c842e4b9599c0be7c7c6db (127.0.98.67:45817): tablet e1758261341e4c9f8badabc05371dbf6 (table test-table [id=66ac27aad5b3453c94b7998615db3841]) successfully deleted
I20260926 20:44:08.590461 7853 ts_tablet_manager.cc:1939] T e1758261341e4c9f8badabc05371dbf6 P a7236e5c160a440aa70f7e343d1623ed: Deleting consensus metadata
I20260926 20:44:08.590767 7258 catalog_manager.cc:4841] TS a7236e5c160a440aa70f7e343d1623ed (127.0.98.66:42795): tablet e1758261341e4c9f8badabc05371dbf6 (table test-table [id=66ac27aad5b3453c94b7998615db3841]) successfully deleted
I20260926 20:44:08.592603 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:08.592496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:36235 (local address 127.0.0.1:55692)
0926 20:44:08.592536 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:08.592566 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:36235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260926 20:44:08.592777 7261 catalog_manager.cc:4584] TS 789f62973fe7453a87bc5bd18132ba42 (127.0.98.65:36235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1758261341e4c9f8badabc05371dbf6: Network error: Client connection negotiation failed: client connection to 127.0.98.65:36235: connect: Connection refused (error 111)
I20260926 20:44:08.592945 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 7462
I20260926 20:44:08.599706 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 7598
I20260926 20:44:08.604430 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 6955
I20260926 20:44:08.609277 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 7143
2026-09-26T20:44:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6507 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:44:08 dist-test-slave-4hd3 krb5kdc[7858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:44:08 dist-test-slave-4hd3 krb5kdc[7858](info): set up 2 sockets
Sep 26 20:44:08 dist-test-slave-4hd3 krb5kdc[7858](info): commencing operation
krb5kdc: starting...
W20260926 20:44:09.636184 393 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:44:09 dist-test-slave-4hd3 krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:44:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:44:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/oryx_127.0.98.126.keytab.
Entry for principal oryx/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/oryx_127.0.98.126.keytab.
I20260926 20:44:09.754853 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37799
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/oryx_127.0.98.126.keytab
--principal=oryx/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:37799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:09.835819 7871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:09.836067 7871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:09.836090 7871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:09.837294 7871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:44:09.837339 7871 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:44:09.837353 7871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:09.837366 7871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:09.837378 7871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:44:09.837390 7871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:44:09.838847 7871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:37799
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/oryx_127.0.98.126.keytab
--principal=oryx/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:09.839102 7871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:09.839339 7871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:09.844029 7877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:09.843997 7876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:09.844170 7871 server_base.cc:1034] running on GCE node
W20260926 20:44:09.844239 7879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:09.844509 7871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:09.844797 7871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:09.845935 7871 hybrid_clock.cc:648] HybridClock initialized: now 1790455449845937 us; error 44 us; skew 500 ppm
Sep 26 20:44:09 dist-test-slave-4hd3 krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455449, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:09.848376 7871 init.cc:377] Logged in from keytab as oryx/127.0.98.126@KRBTEST.COM (short username oryx)
I20260926 20:44:09.848799 7871 webserver.cc:460] Webserver started at http://127.0.98.126:38051/ using document root <none> and password file <none>
I20260926 20:44:09.849030 7871 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:09.849077 7871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:09.849242 7871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:09.850262 7871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "3786db80b7ba4acfb9fbb81abbb40a79"
format_stamp: "Formatted at 2026-09-26 20:44:09 on dist-test-slave-4hd3"
server_key: "23d2f05f0a01bd6f15cc39890c9f6f37"
server_key_iv: "61efdfa0341a23009a979eefb6834231"
server_key_version: "encryptionkey@0"
I20260926 20:44:09.850618 7871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "3786db80b7ba4acfb9fbb81abbb40a79"
format_stamp: "Formatted at 2026-09-26 20:44:09 on dist-test-slave-4hd3"
server_key: "23d2f05f0a01bd6f15cc39890c9f6f37"
server_key_iv: "61efdfa0341a23009a979eefb6834231"
server_key_version: "encryptionkey@0"
I20260926 20:44:09.851904 7871 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:44:09.852618 7886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:09.852787 7871 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:09.852882 7871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "3786db80b7ba4acfb9fbb81abbb40a79"
format_stamp: "Formatted at 2026-09-26 20:44:09 on dist-test-slave-4hd3"
server_key: "23d2f05f0a01bd6f15cc39890c9f6f37"
server_key_iv: "61efdfa0341a23009a979eefb6834231"
server_key_version: "encryptionkey@0"
I20260926 20:44:09.852958 7871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:09.884788 7871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:09.886943 7871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:09.887141 7871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:09.891812 7871 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:37799
I20260926 20:44:09.891870 7948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:37799 every 8 connection(s)
I20260926 20:44:09.892241 7871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:44:09.892912 7949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:09.895076 7949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79: Bootstrap starting.
I20260926 20:44:09.895591 7949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:09.895923 7949 log.cc:824] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79: Log is configured to *not* fsync() on all Append() calls
I20260926 20:44:09.896471 7949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79: No bootstrap required, opened a new log
I20260926 20:44:09.898450 7949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3786db80b7ba4acfb9fbb81abbb40a79" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37799 } }
I20260926 20:44:09.898586 7949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:09.898607 7949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3786db80b7ba4acfb9fbb81abbb40a79, State: Initialized, Role: FOLLOWER
I20260926 20:44:09.898741 7949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3786db80b7ba4acfb9fbb81abbb40a79" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37799 } }
I20260926 20:44:09.898818 7949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:44:09.898840 7949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:44:09.898867 7949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:09.899333 7949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3786db80b7ba4acfb9fbb81abbb40a79" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37799 } }
I20260926 20:44:09.899444 7949 leader_election.cc:302] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3786db80b7ba4acfb9fbb81abbb40a79; no voters:
I20260926 20:44:09.899590 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 7871
I20260926 20:44:09.899616 7949 leader_election.cc:288] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:44:09.899685 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:44:09.899684 7952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:44:09.899849 7952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79 [term 1 LEADER]: Becoming Leader. State: Replica: 3786db80b7ba4acfb9fbb81abbb40a79, State: Running, Role: LEADER
I20260926 20:44:09.899899 7949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:44:09.899919 393 external_mini_cluster.cc:1428] Setting key 09f8da75202b97453fe613a326b5451d
I20260926 20:44:09.899943 7952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3786db80b7ba4acfb9fbb81abbb40a79" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37799 } }
I20260926 20:44:09.900272 7952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3786db80b7ba4acfb9fbb81abbb40a79. Latest consensus state: current_term: 1 leader_uuid: "3786db80b7ba4acfb9fbb81abbb40a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3786db80b7ba4acfb9fbb81abbb40a79" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37799 } } }
I20260926 20:44:09.900365 7952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79 [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:09.900281 7954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3786db80b7ba4acfb9fbb81abbb40a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3786db80b7ba4acfb9fbb81abbb40a79" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37799 } } }
I20260926 20:44:09.900516 7954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79 [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:09.900882 7959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:44:09.901408 7959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:44:09.902949 7959 catalog_manager.cc:1347] Generated new cluster ID: 9cb891b03cf1489c9ee6d9a405b57339
I20260926 20:44:09.903012 7959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:44:09 dist-test-slave-4hd3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260926 20:44:09.911468 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:09.900549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42578 (local address 127.0.98.126:37799)
0926 20:44:09.900691 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:09.900702 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:09.900737 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:44:09.900858 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:09.900865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:09.900923 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:09.901696 (+ 773us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:09.901893 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:09.903271 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:09.904326 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:09.904955 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:09.905069 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:09.907854 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:09.907858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:09.907858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:09.907869 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:09.909802 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:09.910203 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:09.910206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:09.910206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:09.910303 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:09.910521 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:09.910522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:09.910522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:09.910809 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:09.910960 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:09.911091 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 20:44:09.911227 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
I20260926 20:44:09.911798 7959 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:44:09.912362 7959 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:44:09.918032 7959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79: Generated new TSK 0
I20260926 20:44:09.918197 7959 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/oryx_127.0.98.65.keytab.
Entry for principal oryx/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/oryx_127.0.98.65.keytab.
I20260926 20:44:09.946604 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37799
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:10.027936 7975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:10.028149 7975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:10.028173 7975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:10.029277 7975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:10.029310 7975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:10.029330 7975 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:44:10.029346 7975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:44:10.030828 7975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:37799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:10.031024 7975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:10.031200 7975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:10.036294 7980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:10.036422 7975 server_base.cc:1034] running on GCE node
W20260926 20:44:10.036341 7981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:10.036294 7983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:10.036772 7975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:10.037025 7975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:10.038187 7975 hybrid_clock.cc:648] HybridClock initialized: now 1790455450038182 us; error 35 us; skew 500 ppm
Sep 26 20:44:10 dist-test-slave-4hd3 krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:10.040704 7975 init.cc:377] Logged in from keytab as oryx/127.0.98.65@KRBTEST.COM (short username oryx)
I20260926 20:44:10.041160 7975 webserver.cc:460] Webserver started at http://127.0.98.65:40045/ using document root <none> and password file <none>
I20260926 20:44:10.041379 7975 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:10.041447 7975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:10.041590 7975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:10.042627 7975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "7aadd954987147dfaff804e8fd158b54"
format_stamp: "Formatted at 2026-09-26 20:44:10 on dist-test-slave-4hd3"
server_key: "65bad9153f455335dfbdf8ba9dd7f893"
server_key_iv: "5ef9881bfd2e16da38c6f064f9433af5"
server_key_version: "encryptionkey@0"
I20260926 20:44:10.042991 7975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "7aadd954987147dfaff804e8fd158b54"
format_stamp: "Formatted at 2026-09-26 20:44:10 on dist-test-slave-4hd3"
server_key: "65bad9153f455335dfbdf8ba9dd7f893"
server_key_iv: "5ef9881bfd2e16da38c6f064f9433af5"
server_key_version: "encryptionkey@0"
I20260926 20:44:10.044401 7975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:44:10.045110 7990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:10.045397 7975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:44:10.045504 7975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "7aadd954987147dfaff804e8fd158b54"
format_stamp: "Formatted at 2026-09-26 20:44:10 on dist-test-slave-4hd3"
server_key: "65bad9153f455335dfbdf8ba9dd7f893"
server_key_iv: "5ef9881bfd2e16da38c6f064f9433af5"
server_key_version: "encryptionkey@0"
I20260926 20:44:10.045578 7975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:10.066637 7975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:10.068800 7975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:10.068977 7975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:10.069497 7975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:10.069554 7975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:10.069597 7975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:10.069648 7975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:44:10 dist-test-slave-4hd3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260926 20:44:10.076752 7975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:35245
I20260926 20:44:10.076800 8105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:35245 every 8 connection(s)
I20260926 20:44:10.077198 7975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:44:10.079295 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.070103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:53279 (local address 127.0.98.126:37799)
0926 20:44:10.070151 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.070153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.070372 (+ 219us) server_negotiation.cc:407] Connection header received
0926 20:44:10.070922 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.070923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.070936 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.070983 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:10.071921 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.072768 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.073893 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.074129 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.074219 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.076670 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:10.076671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:10.076672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:10.076682 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:10.078257 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:10.078602 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:10.078603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:10.078603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:10.078653 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:10.078817 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:10.078818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:10.078818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:10.078909 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:10.078957 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.079173 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.079233 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:44:10.079442 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.070096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37799 (local address 127.0.98.65:53279)
0926 20:44:10.070249 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:10.070284 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:10.070811 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:10.071057 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:10.071062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:10.071464 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:10.071787 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.071793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.072866 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.072867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.073799 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.073801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.074238 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.074239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.074355 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.074882 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:10.074891 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:10.076556 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:10.078325 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:10.078329 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:10.078330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:10.078537 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:10.078693 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:10.078694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:10.078694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:10.078771 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:10.078965 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:10.078969 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:10.079073 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 20:44:10.079152 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:44:10.079261 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":90,"threads_started":1}
I20260926 20:44:10.079919 8106 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37799
I20260926 20:44:10.080036 8106 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:10.080214 8106 heartbeater.cc:511] Master 127.0.98.126:37799 requested a full tablet report, sending...
I20260926 20:44:10.080749 7902 ts_manager.cc:194] Registered new tserver with Master: 7aadd954987147dfaff804e8fd158b54 (127.0.98.65:35245)
I20260926 20:44:10.081606 7902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:53279
I20260926 20:44:10.081943 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 7975
I20260926 20:44:10.082036 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:44:10.082242 393 external_mini_cluster.cc:1428] Setting key 4f90f33f156f791ff597d290b7fdd2b9
I20260926 20:44:10.086830 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.080916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:57939 (local address 127.0.98.126:37799)
0926 20:44:10.080961 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.080963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.081130 (+ 167us) server_negotiation.cc:407] Connection header received
0926 20:44:10.081151 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.081152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.081161 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.081214 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:10.081473 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.082150 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.083170 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.083375 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.083461 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.084340 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:10.084341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:10.084341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:10.084347 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:10.085695 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:10.086053 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:10.086054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:10.086055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:10.086111 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:10.086313 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:10.086314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:10.086314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:10.086439 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:10.086504 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.086674 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.086745 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 20:44:10.086838 8112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.080858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37799 (local address 127.0.98.65:57939)
0926 20:44:10.081035 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:10.081044 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:10.081093 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:10.081225 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:10.081225 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:10.081357 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:10.081413 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.081414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.082239 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.082240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.083065 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.083067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.083462 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.083463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.083525 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.083968 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:10.083972 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:10.084236 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:10.085796 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:10.085797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:10.085798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:10.085969 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:10.086184 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:10.086185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:10.086186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:10.086242 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:10.086530 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:10.086530 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:10.086575 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 20:44:10.086651 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:44:10.086745 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":135,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/oryx_127.0.98.66.keytab.
Entry for principal oryx/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/oryx_127.0.98.66.keytab.
I20260926 20:44:10.107184 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37799
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:10.186800 8115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:10.187004 8115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:10.187027 8115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:10.188153 8115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:10.188192 8115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:10.188212 8115 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:44:10.188230 8115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:44:10.189617 8115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:37799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:10.189793 8115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:10.189980 8115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:10.194947 8121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:10.194990 8123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:10.194981 8120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:10.195063 8115 server_base.cc:1034] running on GCE node
I20260926 20:44:10.195348 8115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:10.195601 8115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:10.196771 8115 hybrid_clock.cc:648] HybridClock initialized: now 1790455450196758 us; error 43 us; skew 500 ppm
Sep 26 20:44:10 dist-test-slave-4hd3 krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:10.199302 8115 init.cc:377] Logged in from keytab as oryx/127.0.98.66@KRBTEST.COM (short username oryx)
I20260926 20:44:10.199775 8115 webserver.cc:460] Webserver started at http://127.0.98.66:44415/ using document root <none> and password file <none>
I20260926 20:44:10.200021 8115 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:10.200088 8115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:10.200220 8115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:10.201332 8115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f"
format_stamp: "Formatted at 2026-09-26 20:44:10 on dist-test-slave-4hd3"
server_key: "072f297c6f431bb11286f9be0bba59c8"
server_key_iv: "b6b14c9549fff559f40bf0e408beb924"
server_key_version: "encryptionkey@0"
I20260926 20:44:10.201685 8115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f"
format_stamp: "Formatted at 2026-09-26 20:44:10 on dist-test-slave-4hd3"
server_key: "072f297c6f431bb11286f9be0bba59c8"
server_key_iv: "b6b14c9549fff559f40bf0e408beb924"
server_key_version: "encryptionkey@0"
I20260926 20:44:10.203114 8115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:44:10.204015 8130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:10.204226 8115 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:10.204353 8115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f"
format_stamp: "Formatted at 2026-09-26 20:44:10 on dist-test-slave-4hd3"
server_key: "072f297c6f431bb11286f9be0bba59c8"
server_key_iv: "b6b14c9549fff559f40bf0e408beb924"
server_key_version: "encryptionkey@0"
I20260926 20:44:10.204437 8115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:10.216650 8115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:10.218924 8115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:10.219082 8115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:10.219574 8115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:10.219633 8115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:10.219686 8115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:10.219730 8115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:44:10 dist-test-slave-4hd3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260926 20:44:10.227165 8115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:46421
I20260926 20:44:10.227242 8245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:46421 every 8 connection(s)
I20260926 20:44:10.227607 8115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:44:10.229588 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.220157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:46151 (local address 127.0.98.126:37799)
0926 20:44:10.220236 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.220240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.220454 (+ 214us) server_negotiation.cc:407] Connection header received
0926 20:44:10.220902 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.220904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.220917 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.220997 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:10.221893 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.222722 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.224498 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.224681 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.224768 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.226956 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:10.226957 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:10.226957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:10.226966 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:10.228466 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:10.228810 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:10.228811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:10.228811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:10.228865 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:10.229098 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:10.229098 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:10.229098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:10.229193 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:10.229238 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.229469 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.229526 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:44:10.229730 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.220086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37799 (local address 127.0.98.66:46151)
0926 20:44:10.220327 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:10.220355 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:10.220806 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:10.221159 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:10.221163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:10.221405 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:10.221765 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.221770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.222851 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.222851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.224383 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.224387 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.224790 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.224791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.224935 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.225439 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:10.225447 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:10.226838 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:10.228538 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:10.228542 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:10.228543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:10.228744 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:10.228964 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:10.228965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:10.228965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:10.229056 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:10.229242 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:10.229246 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:10.229357 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 20:44:10.229439 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:44:10.229555 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260926 20:44:10.230262 8246 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37799
I20260926 20:44:10.230345 8246 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:10.230552 8246 heartbeater.cc:511] Master 127.0.98.126:37799 requested a full tablet report, sending...
I20260926 20:44:10.230927 7902 ts_manager.cc:194] Registered new tserver with Master: f4bf4e6e1c474e2aa7da9b99d5c1df3f (127.0.98.66:46421)
I20260926 20:44:10.231423 7902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:46151
I20260926 20:44:10.231791 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 8115
I20260926 20:44:10.231906 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:44:10.232113 393 external_mini_cluster.cc:1428] Setting key 2d0503564569319b38acd394219073e2
I20260926 20:44:10.237272 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.231379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:48235 (local address 127.0.98.126:37799)
0926 20:44:10.231422 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.231424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.231482 (+ 58us) server_negotiation.cc:407] Connection header received
0926 20:44:10.231563 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.231564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.231574 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.231630 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:10.232095 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.232783 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.233498 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.233701 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.233774 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.234668 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:10.234669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:10.234669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:10.234676 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:10.236130 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:10.236482 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:10.236483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:10.236483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:10.236535 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:10.236744 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:10.236745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:10.236745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:10.236879 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:10.236957 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.237113 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.237186 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 20:44:10.237289 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.231307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37799 (local address 127.0.98.66:48235)
0926 20:44:10.231428 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:10.231439 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:10.231498 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:10.231680 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:10.231681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:10.231928 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:10.232005 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.232007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.232878 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.232879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.233402 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.233403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.233772 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.233773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.233842 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.234282 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:10.234287 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:10.234563 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:10.236222 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:10.236223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:10.236224 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:10.236398 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:10.236607 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:10.236608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:10.236609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:10.236665 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:10.236962 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:10.236963 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:10.237006 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 20:44:10.237090 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:44:10.237187 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/oryx_127.0.98.67.keytab.
Entry for principal oryx/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/oryx_127.0.98.67.keytab.
I20260926 20:44:10.259927 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37799
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:10.341040 8255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:10.341284 8255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:10.341321 8255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:10.342489 8255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:10.342559 8255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:10.342592 8255 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:44:10.342633 8255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:44:10.344184 8255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:37799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:10.344458 8255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:10.344694 8255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:44:10.349699 8255 server_base.cc:1034] running on GCE node
W20260926 20:44:10.349684 8263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:10.349684 8261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:10.349730 8260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:10.350013 8255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:10.350276 8255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:10.351428 8255 hybrid_clock.cc:648] HybridClock initialized: now 1790455450351405 us; error 30 us; skew 500 ppm
Sep 26 20:44:10 dist-test-slave-4hd3 krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:10.354705 8255 init.cc:377] Logged in from keytab as oryx/127.0.98.67@KRBTEST.COM (short username oryx)
I20260926 20:44:10.355289 8255 webserver.cc:460] Webserver started at http://127.0.98.67:40213/ using document root <none> and password file <none>
I20260926 20:44:10.355540 8255 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:10.355613 8255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:10.355774 8255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:10.357100 8255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "f5fa6c406b5548a79ad3dd8a95abb65a"
format_stamp: "Formatted at 2026-09-26 20:44:10 on dist-test-slave-4hd3"
server_key: "25125398d649798ab3597a487f521a30"
server_key_iv: "d1aac715d89fd37c871783fe989b9236"
server_key_version: "encryptionkey@0"
I20260926 20:44:10.357468 8255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "f5fa6c406b5548a79ad3dd8a95abb65a"
format_stamp: "Formatted at 2026-09-26 20:44:10 on dist-test-slave-4hd3"
server_key: "25125398d649798ab3597a487f521a30"
server_key_iv: "d1aac715d89fd37c871783fe989b9236"
server_key_version: "encryptionkey@0"
I20260926 20:44:10.358816 8255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:44:10.359598 8270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:10.359824 8255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:10.359941 8255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "f5fa6c406b5548a79ad3dd8a95abb65a"
format_stamp: "Formatted at 2026-09-26 20:44:10 on dist-test-slave-4hd3"
server_key: "25125398d649798ab3597a487f521a30"
server_key_iv: "d1aac715d89fd37c871783fe989b9236"
server_key_version: "encryptionkey@0"
I20260926 20:44:10.360026 8255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:10.374732 8255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:10.377142 8255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:10.377305 8255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:10.377815 8255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:10.377854 8255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:10.377878 8255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:10.377892 8255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:10.384804 8255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:38551
Sep 26 20:44:10 dist-test-slave-4hd3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260926 20:44:10.384904 8385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:38551 every 8 connection(s)
I20260926 20:44:10.385206 8255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:44:10.387910 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.378341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:40447 (local address 127.0.98.126:37799)
0926 20:44:10.378416 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.378419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.378614 (+ 195us) server_negotiation.cc:407] Connection header received
0926 20:44:10.379218 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.379220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.379235 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.379315 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:10.380158 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.381017 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.382066 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.382260 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.382358 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.385455 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:10.385456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:10.385457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:10.385465 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:10.386659 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:10.387047 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:10.387048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:10.387048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:10.387105 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:10.387327 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:10.387328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:10.387328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:10.387438 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:10.387491 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.387767 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.387844 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:44:10.388062 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.378321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37799 (local address 127.0.98.67:40447)
0926 20:44:10.378481 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:10.378518 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:10.379101 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:10.379428 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:10.379432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:10.379808 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:10.380067 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.380072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.381150 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.381151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.381971 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.381973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.382391 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.382392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.382514 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.383654 (+ 1140us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:10.383662 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:10.385336 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:10.386729 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:10.386734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:10.386735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:10.386947 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:10.387173 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:10.387173 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:10.387174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:10.387267 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:10.387494 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:10.387498 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:10.387609 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 20:44:10.387688 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:44:10.387874 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260926 20:44:10.388497 8386 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37799
I20260926 20:44:10.388599 8386 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:10.388772 8386 heartbeater.cc:511] Master 127.0.98.126:37799 requested a full tablet report, sending...
I20260926 20:44:10.389119 7903 ts_manager.cc:194] Registered new tserver with Master: f5fa6c406b5548a79ad3dd8a95abb65a (127.0.98.67:38551)
I20260926 20:44:10.389456 7903 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:40447
I20260926 20:44:10.394408 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.389562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:52879 (local address 127.0.98.126:37799)
0926 20:44:10.389597 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.389599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.389665 (+ 66us) server_negotiation.cc:407] Connection header received
0926 20:44:10.389730 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.389731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.389741 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.389794 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:10.390115 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.390568 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.391252 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.391449 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.391529 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.392305 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:10.392307 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:10.392307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:10.392313 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:10.393337 (+ 1024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:10.393658 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:10.393659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:10.393660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:10.393716 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:10.393934 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:10.393935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:10.393936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:10.394042 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:10.394117 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.394257 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.394325 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:44:10.394428 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.389501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37799 (local address 127.0.98.67:52879)
0926 20:44:10.389608 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:10.389619 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:10.389675 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:10.389811 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:10.389812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:10.389982 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:10.390029 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.390030 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.390658 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.390658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.391159 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.391161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.391532 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.391533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.391594 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.391972 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:10.391976 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:10.392221 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:10.393420 (+ 1199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:10.393421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:10.393421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:10.393589 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:10.393782 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:10.393783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:10.393783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:10.393838 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:10.394108 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:10.394109 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:10.394151 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 20:44:10.394234 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 20:44:10.394328 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":61,"threads_started":1}
I20260926 20:44:10.394852 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 8255
I20260926 20:44:10.394956 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:44:10.395174 393 external_mini_cluster.cc:1428] Setting key 0f3879b2fc6353a0997350625578301a
I20260926 20:44:10.396130 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:44:10 dist-test-slave-4hd3 krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455450, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:44:10 dist-test-slave-4hd3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:10 dist-test-slave-4hd3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:44:10.416880 7963 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:10.411557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42590 (local address 127.0.98.126:37799)
0926 20:44:10.411622 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.411624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.411676 (+ 52us) server_negotiation.cc:407] Connection header received
0926 20:44:10.411762 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.411763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.411775 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.411837 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:10.412330 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.412792 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.413549 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.413719 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.413769 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.416804 (+ 3035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42590: BlockingRecv error: recv got EOF from 127.0.0.1:42590 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 26 20:44:10 dist-test-slave-4hd3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455450, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260926 20:44:10.423966 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.417694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42594 (local address 127.0.98.126:37799)
0926 20:44:10.417736 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.417738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.417839 (+ 101us) server_negotiation.cc:407] Connection header received
0926 20:44:10.417896 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.417897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.417905 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.417945 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:10.418471 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.418920 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.419563 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.419790 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.419850 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.421909 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:10.421910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:10.421910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:10.421916 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:10.423047 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:10.423332 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:10.423332 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:10.423333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:10.423379 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:10.423522 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:10.423523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:10.423523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:10.423618 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:10.423688 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.423830 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.423884 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 20:44:10.425421 7903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:44:10.426923 7903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:44:10.433225 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.429268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46421 (local address 127.0.0.1:57622)
0926 20:44:10.429447 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:10.429489 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:10.429548 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:10.429779 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:10.429782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:10.429784 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:10.429847 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.429848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.430931 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.430932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.431929 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.431931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.432926 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.432927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.433013 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.433027 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 20:44:10.433104 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:44:10.433152 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":41,"threads_started":1}
I20260926 20:44:10.433256 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.429357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57622 (local address 127.0.98.66:46421)
0926 20:44:10.429552 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.429557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.429569 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:44:10.429658 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.429660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.429679 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.429767 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:10.429926 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.430836 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.432011 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.432838 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.432924 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.433002 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.433127 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.433162 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":97,"threads_started":1}
I20260926 20:44:10.433651 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.429496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:35245 (local address 127.0.0.1:45272)
0926 20:44:10.429898 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:10.429909 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:10.429958 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:10.430322 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:10.430322 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:10.430324 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:10.430366 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.430369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.431231 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.431232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.432572 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.432574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.433432 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.433433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.433487 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.433490 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:44:10.433550 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 20:44:10.433587 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":95,"threads_started":1}
I20260926 20:44:10.433699 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.429898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45272 (local address 127.0.98.65:35245)
0926 20:44:10.430115 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.430120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.430130 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:44:10.430168 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.430170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.430186 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.430323 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:10.430441 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.431104 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.432658 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.433359 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.433431 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.433510 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.433565 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.433599 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":72,"threads_started":1}
I20260926 20:44:10.434188 8180 tablet_service.cc:1467] Processing CreateTablet for tablet b5c2f9701b3f4f20b6b9a6c4edc25a27 (DEFAULT_TABLE table=test-table [id=a9e23f6ae1cc44b9a1cb29975925cf8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:10.434259 8040 tablet_service.cc:1467] Processing CreateTablet for tablet b5c2f9701b3f4f20b6b9a6c4edc25a27 (DEFAULT_TABLE table=test-table [id=a9e23f6ae1cc44b9a1cb29975925cf8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:10.434562 8180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5c2f9701b3f4f20b6b9a6c4edc25a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:10.434612 8040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5c2f9701b3f4f20b6b9a6c4edc25a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:10.434753 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.429291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:38551 (local address 127.0.0.1:38228)
0926 20:44:10.429644 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:10.429654 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:10.429693 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:10.430311 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:10.430312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:10.430314 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:10.430357 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.430359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.432462 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.432463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.433547 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.433550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.434529 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.434530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.434596 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.434600 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:44:10.434656 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 20:44:10.434694 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":95,"threads_started":1}
I20260926 20:44:10.434820 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.429654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38228 (local address 127.0.98.67:38551)
0926 20:44:10.430106 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.430110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.430120 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:44:10.430155 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.430157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.430172 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.430242 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:10.431258 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.432386 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.433648 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.434345 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.434424 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.434494 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.434691 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.434732 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":82,"threads_started":1}
I20260926 20:44:10.435388 8320 tablet_service.cc:1467] Processing CreateTablet for tablet b5c2f9701b3f4f20b6b9a6c4edc25a27 (DEFAULT_TABLE table=test-table [id=a9e23f6ae1cc44b9a1cb29975925cf8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:10.435684 8320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5c2f9701b3f4f20b6b9a6c4edc25a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:10.437850 8414 tablet_bootstrap.cc:492] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54: Bootstrap starting.
I20260926 20:44:10.438442 8414 tablet_bootstrap.cc:654] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:10.438457 8415 tablet_bootstrap.cc:492] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Bootstrap starting.
I20260926 20:44:10.438767 8414 log.cc:824] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54: Log is configured to *not* fsync() on all Append() calls
I20260926 20:44:10.439071 8415 tablet_bootstrap.cc:654] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:10.439344 8417 tablet_bootstrap.cc:492] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a: Bootstrap starting.
I20260926 20:44:10.439373 8415 log.cc:824] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:44:10.439983 8417 tablet_bootstrap.cc:654] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:10.440075 8414 tablet_bootstrap.cc:492] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54: No bootstrap required, opened a new log
I20260926 20:44:10.440151 8415 tablet_bootstrap.cc:492] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: No bootstrap required, opened a new log
I20260926 20:44:10.440152 8414 ts_tablet_manager.cc:1397] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:44:10.440217 8415 ts_tablet_manager.cc:1397] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:44:10.440325 8417 log.cc:824] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:44:10.441190 8417 tablet_bootstrap.cc:492] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a: No bootstrap required, opened a new log
I20260926 20:44:10.441259 8417 ts_tablet_manager.cc:1397] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:44:10.441926 8414 raft_consensus.cc:348] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } }
I20260926 20:44:10.442081 8414 raft_consensus.cc:374] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:10.442123 8414 raft_consensus.cc:729] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aadd954987147dfaff804e8fd158b54, State: Initialized, Role: FOLLOWER
I20260926 20:44:10.442255 8414 consensus_queue.cc:260] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } }
I20260926 20:44:10.442407 8415 raft_consensus.cc:348] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } }
I20260926 20:44:10.442533 8106 heartbeater.cc:503] Master 127.0.98.126:37799 was elected leader, sending a full tablet report...
I20260926 20:44:10.442564 8415 raft_consensus.cc:374] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:10.442533 8414 ts_tablet_manager.cc:1428] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:44:10.442600 8415 raft_consensus.cc:729] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4bf4e6e1c474e2aa7da9b99d5c1df3f, State: Initialized, Role: FOLLOWER
I20260926 20:44:10.442720 8415 consensus_queue.cc:260] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } }
I20260926 20:44:10.442961 8415 ts_tablet_manager.cc:1428] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:44:10.443075 8246 heartbeater.cc:503] Master 127.0.98.126:37799 was elected leader, sending a full tablet report...
I20260926 20:44:10.443228 8417 raft_consensus.cc:348] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } }
I20260926 20:44:10.443372 8417 raft_consensus.cc:374] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:10.443423 8417 raft_consensus.cc:729] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fa6c406b5548a79ad3dd8a95abb65a, State: Initialized, Role: FOLLOWER
I20260926 20:44:10.443535 8417 consensus_queue.cc:260] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } }
I20260926 20:44:10.443799 8417 ts_tablet_manager.cc:1428] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:44:10.443921 8386 heartbeater.cc:503] Master 127.0.98.126:37799 was elected leader, sending a full tablet report...
I20260926 20:44:10.466040 8421 raft_consensus.cc:482] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:44:10.466200 8421 raft_consensus.cc:504] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } }
I20260926 20:44:10.466540 8421 leader_election.cc:288] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5fa6c406b5548a79ad3dd8a95abb65a (127.0.98.67:38551), 7aadd954987147dfaff804e8fd158b54 (127.0.98.65:35245)
I20260926 20:44:10.469571 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.466807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:37259 (local address 127.0.98.67:38551)
0926 20:44:10.466868 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.466870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.466881 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:44:10.467036 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.467037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.467051 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.467110 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:10.467300 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.467839 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.468813 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.469201 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.469282 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.469292 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.469457 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.469493 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:44:10.469581 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.466952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:41629 (local address 127.0.98.65:35245)
0926 20:44:10.467003 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.467006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.467119 (+ 113us) server_negotiation.cc:407] Connection header received
0926 20:44:10.467151 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.467152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.467164 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.467228 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:10.467397 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.467950 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.468845 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.469202 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.469286 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.469296 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.469476 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.469510 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 20:44:10.469606 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.466736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:38551 (local address 127.0.98.66:37259)
0926 20:44:10.466796 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:10.466814 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:10.466885 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:10.467139 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:10.467141 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:10.467143 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:10.467225 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.467227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.467912 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.467913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.468716 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.468719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.469295 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.469296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.469384 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.469387 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:44:10.469427 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 20:44:10.469493 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 20:44:10.469813 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.466861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:35245 (local address 127.0.98.66:41629)
0926 20:44:10.467042 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:10.467053 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:10.467108 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:10.467238 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:10.467238 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:10.467239 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:10.467322 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.467324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.468031 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.468032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.468733 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.468735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.469288 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.469289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.469379 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.469382 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:44:10.469459 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:44:10.469498 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":9,"thread_start_us":131,"threads_started":1}
I20260926 20:44:10.469961 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c2f9701b3f4f20b6b9a6c4edc25a27" candidate_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" is_pre_election: true
I20260926 20:44:10.470134 8340 raft_consensus.cc:2454] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4bf4e6e1c474e2aa7da9b99d5c1df3f in term 0.
I20260926 20:44:10.470412 8134 leader_election.cc:302] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f4bf4e6e1c474e2aa7da9b99d5c1df3f, f5fa6c406b5548a79ad3dd8a95abb65a; no voters:
I20260926 20:44:10.470515 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c2f9701b3f4f20b6b9a6c4edc25a27" candidate_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aadd954987147dfaff804e8fd158b54" is_pre_election: true
I20260926 20:44:10.470561 8421 raft_consensus.cc:2781] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:44:10.470628 8421 raft_consensus.cc:482] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:44:10.470662 8421 raft_consensus.cc:3037] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:10.470685 8060 raft_consensus.cc:2454] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4bf4e6e1c474e2aa7da9b99d5c1df3f in term 0.
I20260926 20:44:10.471674 8421 raft_consensus.cc:504] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } }
I20260926 20:44:10.471841 8421 leader_election.cc:288] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [CANDIDATE]: Term 1 election: Requested vote from peers f5fa6c406b5548a79ad3dd8a95abb65a (127.0.98.67:38551), 7aadd954987147dfaff804e8fd158b54 (127.0.98.65:35245)
I20260926 20:44:10.471989 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c2f9701b3f4f20b6b9a6c4edc25a27" candidate_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a"
I20260926 20:44:10.472010 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c2f9701b3f4f20b6b9a6c4edc25a27" candidate_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aadd954987147dfaff804e8fd158b54"
I20260926 20:44:10.472076 8340 raft_consensus.cc:3037] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:10.472128 8060 raft_consensus.cc:3037] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:10.472779 8340 raft_consensus.cc:2454] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4bf4e6e1c474e2aa7da9b99d5c1df3f in term 1.
I20260926 20:44:10.472822 8060 raft_consensus.cc:2454] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4bf4e6e1c474e2aa7da9b99d5c1df3f in term 1.
I20260926 20:44:10.472987 8134 leader_election.cc:302] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aadd954987147dfaff804e8fd158b54, f4bf4e6e1c474e2aa7da9b99d5c1df3f; no voters:
I20260926 20:44:10.473111 8421 raft_consensus.cc:2781] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:44:10.473304 8421 raft_consensus.cc:686] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 LEADER]: Becoming Leader. State: Replica: f4bf4e6e1c474e2aa7da9b99d5c1df3f, State: Running, Role: LEADER
I20260926 20:44:10.473408 8421 consensus_queue.cc:237] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } }
I20260926 20:44:10.474236 7903 catalog_manager.cc:5495] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f4bf4e6e1c474e2aa7da9b99d5c1df3f (127.0.98.66). New cstate: current_term: 1 leader_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } health_report { overall_health: UNKNOWN } } }
W20260926 20:44:10.478248 8247 tablet.cc:2369] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:44:10.490952 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.488495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57638 (local address 127.0.98.66:46421)
0926 20:44:10.488536 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.488538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.488546 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:44:10.488583 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.488583 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.488602 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.488689 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:44:10.488806 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.489405 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.490172 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.490374 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.490458 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.490624 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:44:10.490725 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:44:10.490809 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.490847 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.490868 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:44:10.492676 8340 raft_consensus.cc:1261] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a [term 1 FOLLOWER]: Refusing update from remote peer f4bf4e6e1c474e2aa7da9b99d5c1df3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:44:10.493045 8060 raft_consensus.cc:1261] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54 [term 1 FOLLOWER]: Refusing update from remote peer f4bf4e6e1c474e2aa7da9b99d5c1df3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:44:10.493027 8421 consensus_queue.cc:1035] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:44:10.493350 8421 consensus_queue.cc:1035] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:44:10.495649 8428 mvcc.cc:205] Tried to move back new op lower bound from 7333705525215387648 to 7333705525139902464. Current Snapshot: MvccSnapshot[applied={T|T < 7333705525215387648}]
I20260926 20:44:10.496397 8429 mvcc.cc:205] Tried to move back new op lower bound from 7333705525215387648 to 7333705525139902464. Current Snapshot: MvccSnapshot[applied={T|T < 7333705525215387648 or (T in {7333705525215387648})}]
I20260926 20:44:10.497473 7903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:44:10.497581 7903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:44:10.498278 7903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79: Sending DeleteTablet for 3 replicas of tablet b5c2f9701b3f4f20b6b9a6c4edc25a27
I20260926 20:44:10.498627 8320 tablet_service.cc:1514] Processing DeleteTablet for tablet b5c2f9701b3f4f20b6b9a6c4edc25a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:10 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:38228
I20260926 20:44:10.498628 8180 tablet_service.cc:1514] Processing DeleteTablet for tablet b5c2f9701b3f4f20b6b9a6c4edc25a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:10 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:57622
I20260926 20:44:10.498627 8040 tablet_service.cc:1514] Processing DeleteTablet for tablet b5c2f9701b3f4f20b6b9a6c4edc25a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:10 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:45272
I20260926 20:44:10.498826 8437 tablet_replica.cc:330] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a: stopping tablet replica
I20260926 20:44:10.498782 7903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 20:44:10.498911 8437 raft_consensus.cc:2229] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 20:44:10.498914 7903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:44:10.499217 8438 tablet_replica.cc:330] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54: stopping tablet replica
I20260926 20:44:10.499230 8437 raft_consensus.cc:2258] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:10.499470 8438 raft_consensus.cc:2229] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:44:10.499671 8437 ts_tablet_manager.cc:1905] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:10.500197 8320 tablet_service.cc:1467] Processing CreateTablet for tablet 4f5f87ff10fb45a3922dd48818b858d7 (DEFAULT_TABLE table=test-table [id=cbf082c837b44a50a0b169ea353e8205]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:10.500223 8040 tablet_service.cc:1467] Processing CreateTablet for tablet 4f5f87ff10fb45a3922dd48818b858d7 (DEFAULT_TABLE table=test-table [id=cbf082c837b44a50a0b169ea353e8205]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:10.500312 8320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f5f87ff10fb45a3922dd48818b858d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:10.500331 8040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f5f87ff10fb45a3922dd48818b858d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:10.501376 8414 tablet_bootstrap.cc:492] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54: Bootstrap starting.
I20260926 20:44:10.501554 8417 tablet_bootstrap.cc:492] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a: Bootstrap starting.
I20260926 20:44:10.501632 8180 tablet_service.cc:1467] Processing CreateTablet for tablet 4f5f87ff10fb45a3922dd48818b858d7 (DEFAULT_TABLE table=test-table [id=cbf082c837b44a50a0b169ea353e8205]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:10.501735 8180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f5f87ff10fb45a3922dd48818b858d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:10.501828 8414 tablet_bootstrap.cc:654] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:10.501945 8417 tablet_bootstrap.cc:654] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:10.502322 8437 ts_tablet_manager.cc:1918] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:44:10.502399 8437 log.cc:1196] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/b5c2f9701b3f4f20b6b9a6c4edc25a27
I20260926 20:44:10.502650 8437 ts_tablet_manager.cc:1939] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f5fa6c406b5548a79ad3dd8a95abb65a: Deleting consensus metadata
I20260926 20:44:10.503062 7890 catalog_manager.cc:4841] TS f5fa6c406b5548a79ad3dd8a95abb65a (127.0.98.67:38551): tablet b5c2f9701b3f4f20b6b9a6c4edc25a27 (table test-table [id=a9e23f6ae1cc44b9a1cb29975925cf8c]) successfully deleted
I20260926 20:44:10.503343 8415 tablet_bootstrap.cc:492] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Bootstrap starting.
I20260926 20:44:10.503353 8414 tablet_bootstrap.cc:492] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54: No bootstrap required, opened a new log
I20260926 20:44:10.503407 8414 ts_tablet_manager.cc:1397] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:44:10.503602 8414 raft_consensus.cc:348] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:10.503696 8414 raft_consensus.cc:374] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:10.503743 8414 raft_consensus.cc:729] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aadd954987147dfaff804e8fd158b54, State: Initialized, Role: FOLLOWER
I20260926 20:44:10.503790 8414 consensus_queue.cc:260] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:10.503856 8417 tablet_bootstrap.cc:492] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a: No bootstrap required, opened a new log
I20260926 20:44:10.503876 8414 ts_tablet_manager.cc:1428] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:10.503904 8417 ts_tablet_manager.cc:1397] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:44:10.503950 8415 tablet_bootstrap.cc:654] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:10.504026 8417 raft_consensus.cc:348] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:10.504091 8417 raft_consensus.cc:374] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:10.504110 8417 raft_consensus.cc:729] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fa6c406b5548a79ad3dd8a95abb65a, State: Initialized, Role: FOLLOWER
I20260926 20:44:10.504153 8417 consensus_queue.cc:260] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:10.504235 8417 ts_tablet_manager.cc:1428] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:10.504416 8438 raft_consensus.cc:2258] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:10.504894 8438 ts_tablet_manager.cc:1905] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:10.504989 8415 tablet_bootstrap.cc:492] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: No bootstrap required, opened a new log
I20260926 20:44:10.505051 8415 ts_tablet_manager.cc:1397] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:44:10.505187 8415 raft_consensus.cc:348] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:10.505254 8415 raft_consensus.cc:374] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:10.505275 8415 raft_consensus.cc:729] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4bf4e6e1c474e2aa7da9b99d5c1df3f, State: Initialized, Role: FOLLOWER
I20260926 20:44:10.505314 8415 consensus_queue.cc:260] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:10.505407 8415 ts_tablet_manager.cc:1428] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 20:44:10.506350 8438 ts_tablet_manager.cc:1918] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:44:10.506407 8438 log.cc:1196] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/wal/wals/b5c2f9701b3f4f20b6b9a6c4edc25a27
I20260926 20:44:10.506699 8438 ts_tablet_manager.cc:1939] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P 7aadd954987147dfaff804e8fd158b54: Deleting consensus metadata
I20260926 20:44:10.507090 7890 catalog_manager.cc:4841] TS 7aadd954987147dfaff804e8fd158b54 (127.0.98.65:35245): tablet b5c2f9701b3f4f20b6b9a6c4edc25a27 (table test-table [id=a9e23f6ae1cc44b9a1cb29975925cf8c]) successfully deleted
I20260926 20:44:10.507841 8436 tablet_replica.cc:330] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: stopping tablet replica
I20260926 20:44:10.507936 8436 raft_consensus.cc:2229] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:44:10.508055 8436 raft_consensus.cc:2258] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:10.508381 8436 ts_tablet_manager.cc:1905] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:10.509553 8436 ts_tablet_manager.cc:1918] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:44:10.509621 8436 log.cc:1196] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/b5c2f9701b3f4f20b6b9a6c4edc25a27
I20260926 20:44:10.509881 8436 ts_tablet_manager.cc:1939] T b5c2f9701b3f4f20b6b9a6c4edc25a27 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Deleting consensus metadata
I20260926 20:44:10.510247 7889 catalog_manager.cc:4841] TS f4bf4e6e1c474e2aa7da9b99d5c1df3f (127.0.98.66:46421): tablet b5c2f9701b3f4f20b6b9a6c4edc25a27 (table test-table [id=a9e23f6ae1cc44b9a1cb29975925cf8c]) successfully deleted
I20260926 20:44:10.517534 8422 raft_consensus.cc:482] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:44:10.517599 8422 raft_consensus.cc:504] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:10.517895 8422 leader_election.cc:288] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7aadd954987147dfaff804e8fd158b54 (127.0.98.65:35245), f4bf4e6e1c474e2aa7da9b99d5c1df3f (127.0.98.66:46421)
I20260926 20:44:10.521454 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.518127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:39167 (local address 127.0.98.66:46421)
0926 20:44:10.518181 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.518183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.518319 (+ 136us) server_negotiation.cc:407] Connection header received
0926 20:44:10.518398 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.518398 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.518406 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.518458 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:10.518625 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.519074 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.520543 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.521109 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.521188 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.521198 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.521350 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.521388 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:44:10.521490 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.518029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46421 (local address 127.0.98.67:39167)
0926 20:44:10.518263 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:10.518273 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:10.518341 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:10.518462 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:10.518463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:10.518464 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:10.518557 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.518558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.519155 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.519156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.520455 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.520457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.521213 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.521214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.521265 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.521269 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:44:10.521343 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:44:10.521386 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":147,"thread_start_us":54,"threads_started":1}
I20260926 20:44:10.521938 8200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f5f87ff10fb45a3922dd48818b858d7" candidate_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" is_pre_election: true
I20260926 20:44:10.522064 8200 raft_consensus.cc:2454] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5fa6c406b5548a79ad3dd8a95abb65a in term 0.
I20260926 20:44:10.522145 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.518026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:35245 (local address 127.0.98.67:51323)
0926 20:44:10.518081 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:10.518091 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:10.518127 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:10.518294 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:10.518295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:10.518297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:10.518358 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.518360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.519252 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.519252 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.520455 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.520457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.521897 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.521898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.521976 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.521979 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:44:10.522035 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 20:44:10.522077 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":28}
I20260926 20:44:10.522177 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.518136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:51323 (local address 127.0.98.65:35245)
0926 20:44:10.518174 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.518177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.518186 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:44:10.518209 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.518210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.518219 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.518292 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:10.518447 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.519173 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.521233 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.521788 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.521856 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.521864 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.522059 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.522104 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:44:10.522446 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f5f87ff10fb45a3922dd48818b858d7" candidate_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aadd954987147dfaff804e8fd158b54" is_pre_election: true
I20260926 20:44:10.522528 8060 raft_consensus.cc:2454] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5fa6c406b5548a79ad3dd8a95abb65a in term 0.
I20260926 20:44:10.522639 8273 leader_election.cc:302] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f4bf4e6e1c474e2aa7da9b99d5c1df3f, f5fa6c406b5548a79ad3dd8a95abb65a; no voters:
I20260926 20:44:10.522785 8422 raft_consensus.cc:2781] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:44:10.522826 8422 raft_consensus.cc:482] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:44:10.522856 8422 raft_consensus.cc:3037] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:10.523454 8422 raft_consensus.cc:504] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:10.523598 8422 leader_election.cc:288] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [CANDIDATE]: Term 1 election: Requested vote from peers 7aadd954987147dfaff804e8fd158b54 (127.0.98.65:35245), f4bf4e6e1c474e2aa7da9b99d5c1df3f (127.0.98.66:46421)
I20260926 20:44:10.523726 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f5f87ff10fb45a3922dd48818b858d7" candidate_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aadd954987147dfaff804e8fd158b54"
I20260926 20:44:10.523789 8060 raft_consensus.cc:3037] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:10.523805 8200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f5f87ff10fb45a3922dd48818b858d7" candidate_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f"
I20260926 20:44:10.523865 8200 raft_consensus.cc:3037] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:10.524392 8200 raft_consensus.cc:2454] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5fa6c406b5548a79ad3dd8a95abb65a in term 1.
I20260926 20:44:10.524461 8060 raft_consensus.cc:2454] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5fa6c406b5548a79ad3dd8a95abb65a in term 1.
I20260926 20:44:10.524617 8274 leader_election.cc:302] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aadd954987147dfaff804e8fd158b54, f5fa6c406b5548a79ad3dd8a95abb65a; no voters:
I20260926 20:44:10.524718 8422 raft_consensus.cc:2781] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:44:10.524847 8422 raft_consensus.cc:686] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [term 1 LEADER]: Becoming Leader. State: Replica: f5fa6c406b5548a79ad3dd8a95abb65a, State: Running, Role: LEADER
I20260926 20:44:10.524909 8422 consensus_queue.cc:237] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:10.525557 7902 catalog_manager.cc:5495] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a reported cstate change: term changed from 0 to 1, leader changed from <none> to f5fa6c406b5548a79ad3dd8a95abb65a (127.0.98.67). New cstate: current_term: 1 leader_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } health_report { overall_health: HEALTHY } } }
Sep 26 20:44:10 dist-test-slave-4hd3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455449, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.126@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260926 20:44:10.551802 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.544398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37799 (local address 127.0.0.1:42610)
0926 20:44:10.544562 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:10.544579 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:10.544659 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:10.544801 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:10.544801 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:10.545123 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:10.545167 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.545169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.545815 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.545816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.546350 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.546352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.546729 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.546730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.546780 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.547393 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:10.547397 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:10.549036 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:10.550629 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:10.550636 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:10.550638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:10.550824 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:10.551053 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:10.551054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:10.551054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:10.551108 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:10.551351 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:10.551356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:10.551405 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 20:44:10.551501 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 20:44:10.551610 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260926 20:44:10.551802 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.544449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42610 (local address 127.0.98.126:37799)
0926 20:44:10.544525 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.544527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.544640 (+ 113us) server_negotiation.cc:407] Connection header received
0926 20:44:10.544718 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.544719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.544731 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.544798 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:10.545261 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.545737 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.546455 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.546630 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.546731 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.549185 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:10.549186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:10.549186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:10.549195 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:10.550535 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:10.550936 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:10.550937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:10.550938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:10.550990 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:10.551179 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:10.551180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:10.551180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:10.551286 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:10.551361 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.551526 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.551658 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":46}
I20260926 20:44:10.554056 7902 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:42610:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:44:10.554275 7902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:44:10.555546 8040 tablet_service.cc:1467] Processing CreateTablet for tablet 785978ea67c04109bb55f967ccda6019 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ed5bc949c624e8e95e922a29324a106]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:44:10.555564 8180 tablet_service.cc:1467] Processing CreateTablet for tablet 785978ea67c04109bb55f967ccda6019 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ed5bc949c624e8e95e922a29324a106]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:44:10.555576 8320 tablet_service.cc:1467] Processing CreateTablet for tablet 785978ea67c04109bb55f967ccda6019 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ed5bc949c624e8e95e922a29324a106]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:44:10.555692 8040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 785978ea67c04109bb55f967ccda6019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:10.555702 8180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 785978ea67c04109bb55f967ccda6019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:10.555702 8320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 785978ea67c04109bb55f967ccda6019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:10.557209 8415 tablet_bootstrap.cc:492] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Bootstrap starting.
I20260926 20:44:10.557219 8414 tablet_bootstrap.cc:492] T 785978ea67c04109bb55f967ccda6019 P 7aadd954987147dfaff804e8fd158b54: Bootstrap starting.
I20260926 20:44:10.557533 8415 tablet_bootstrap.cc:654] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:10.557616 8414 tablet_bootstrap.cc:654] T 785978ea67c04109bb55f967ccda6019 P 7aadd954987147dfaff804e8fd158b54: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:10.557921 8417 tablet_bootstrap.cc:492] T 785978ea67c04109bb55f967ccda6019 P f5fa6c406b5548a79ad3dd8a95abb65a: Bootstrap starting.
I20260926 20:44:10.558094 8415 tablet_bootstrap.cc:492] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: No bootstrap required, opened a new log
I20260926 20:44:10.558125 8415 ts_tablet_manager.cc:1397] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:44:10.558239 8415 raft_consensus.cc:348] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } }
I20260926 20:44:10.558282 8415 raft_consensus.cc:374] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:10.558279 8417 tablet_bootstrap.cc:654] T 785978ea67c04109bb55f967ccda6019 P f5fa6c406b5548a79ad3dd8a95abb65a: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:10.558297 8415 raft_consensus.cc:729] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4bf4e6e1c474e2aa7da9b99d5c1df3f, State: Initialized, Role: FOLLOWER
I20260926 20:44:10.558360 8415 consensus_queue.cc:260] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } }
I20260926 20:44:10.558473 8415 ts_tablet_manager.cc:1428] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:10.558486 8421 tablet_replica.cc:439] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } }
I20260926 20:44:10.558552 8421 tablet_replica.cc:442] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:44:10.558883 8414 tablet_bootstrap.cc:492] T 785978ea67c04109bb55f967ccda6019 P 7aadd954987147dfaff804e8fd158b54: No bootstrap required, opened a new log
I20260926 20:44:10.558979 8414 ts_tablet_manager.cc:1397] T 785978ea67c04109bb55f967ccda6019 P 7aadd954987147dfaff804e8fd158b54: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:44:10.559134 8414 raft_consensus.cc:348] T 785978ea67c04109bb55f967ccda6019 P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } }
I20260926 20:44:10.559190 8414 raft_consensus.cc:374] T 785978ea67c04109bb55f967ccda6019 P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:10.559204 8414 raft_consensus.cc:729] T 785978ea67c04109bb55f967ccda6019 P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aadd954987147dfaff804e8fd158b54, State: Initialized, Role: FOLLOWER
I20260926 20:44:10.559270 8414 consensus_queue.cc:260] T 785978ea67c04109bb55f967ccda6019 P 7aadd954987147dfaff804e8fd158b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } }
I20260926 20:44:10.559355 8414 ts_tablet_manager.cc:1428] T 785978ea67c04109bb55f967ccda6019 P 7aadd954987147dfaff804e8fd158b54: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:44:10.559368 8417 tablet_bootstrap.cc:492] T 785978ea67c04109bb55f967ccda6019 P f5fa6c406b5548a79ad3dd8a95abb65a: No bootstrap required, opened a new log
I20260926 20:44:10.559418 8417 ts_tablet_manager.cc:1397] T 785978ea67c04109bb55f967ccda6019 P f5fa6c406b5548a79ad3dd8a95abb65a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:44:10.559415 8420 tablet_replica.cc:439] T 785978ea67c04109bb55f967ccda6019 P 7aadd954987147dfaff804e8fd158b54: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } }
I20260926 20:44:10.559478 8420 tablet_replica.cc:442] T 785978ea67c04109bb55f967ccda6019 P 7aadd954987147dfaff804e8fd158b54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:44:10.559556 8417 raft_consensus.cc:348] T 785978ea67c04109bb55f967ccda6019 P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } }
I20260926 20:44:10.559612 8417 raft_consensus.cc:374] T 785978ea67c04109bb55f967ccda6019 P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:10.559629 8417 raft_consensus.cc:729] T 785978ea67c04109bb55f967ccda6019 P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fa6c406b5548a79ad3dd8a95abb65a, State: Initialized, Role: FOLLOWER
I20260926 20:44:10.559666 8417 consensus_queue.cc:260] T 785978ea67c04109bb55f967ccda6019 P f5fa6c406b5548a79ad3dd8a95abb65a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } }
I20260926 20:44:10.559777 8417 ts_tablet_manager.cc:1428] T 785978ea67c04109bb55f967ccda6019 P f5fa6c406b5548a79ad3dd8a95abb65a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:10.559818 8422 tablet_replica.cc:439] T 785978ea67c04109bb55f967ccda6019 P f5fa6c406b5548a79ad3dd8a95abb65a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } }
I20260926 20:44:10.559906 8422 tablet_replica.cc:442] T 785978ea67c04109bb55f967ccda6019 P f5fa6c406b5548a79ad3dd8a95abb65a: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 20:44:10.578177 8107 tablet.cc:2369] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:44:10.636034 8387 tablet.cc:2369] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:44:10.867125 8421 raft_consensus.cc:482] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:44:10.867231 8421 raft_consensus.cc:504] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } }
I20260926 20:44:10.867455 8421 leader_election.cc:288] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7aadd954987147dfaff804e8fd158b54 (127.0.98.65:35245), f5fa6c406b5548a79ad3dd8a95abb65a (127.0.98.67:38551)
I20260926 20:44:10.867656 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785978ea67c04109bb55f967ccda6019" candidate_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aadd954987147dfaff804e8fd158b54" is_pre_election: true
I20260926 20:44:10.867789 8060 raft_consensus.cc:2454] T 785978ea67c04109bb55f967ccda6019 P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4bf4e6e1c474e2aa7da9b99d5c1df3f in term 0.
I20260926 20:44:10.867940 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785978ea67c04109bb55f967ccda6019" candidate_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" is_pre_election: true
I20260926 20:44:10.867970 8134 leader_election.cc:302] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aadd954987147dfaff804e8fd158b54, f4bf4e6e1c474e2aa7da9b99d5c1df3f; no voters:
I20260926 20:44:10.868038 8340 raft_consensus.cc:2454] T 785978ea67c04109bb55f967ccda6019 P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4bf4e6e1c474e2aa7da9b99d5c1df3f in term 0.
I20260926 20:44:10.868109 8421 raft_consensus.cc:2781] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:44:10.868162 8421 raft_consensus.cc:482] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:44:10.868186 8421 raft_consensus.cc:3037] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:10.868866 8421 raft_consensus.cc:504] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } }
I20260926 20:44:10.868973 8421 leader_election.cc:288] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [CANDIDATE]: Term 1 election: Requested vote from peers 7aadd954987147dfaff804e8fd158b54 (127.0.98.65:35245), f5fa6c406b5548a79ad3dd8a95abb65a (127.0.98.67:38551)
I20260926 20:44:10.869166 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785978ea67c04109bb55f967ccda6019" candidate_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aadd954987147dfaff804e8fd158b54"
I20260926 20:44:10.869246 8060 raft_consensus.cc:3037] T 785978ea67c04109bb55f967ccda6019 P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:10.869185 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785978ea67c04109bb55f967ccda6019" candidate_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a"
I20260926 20:44:10.869272 8340 raft_consensus.cc:3037] T 785978ea67c04109bb55f967ccda6019 P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:10.869832 8060 raft_consensus.cc:2454] T 785978ea67c04109bb55f967ccda6019 P 7aadd954987147dfaff804e8fd158b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4bf4e6e1c474e2aa7da9b99d5c1df3f in term 1.
I20260926 20:44:10.869978 8340 raft_consensus.cc:2454] T 785978ea67c04109bb55f967ccda6019 P f5fa6c406b5548a79ad3dd8a95abb65a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4bf4e6e1c474e2aa7da9b99d5c1df3f in term 1.
I20260926 20:44:10.869999 8134 leader_election.cc:302] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aadd954987147dfaff804e8fd158b54, f4bf4e6e1c474e2aa7da9b99d5c1df3f; no voters:
I20260926 20:44:10.870091 8421 raft_consensus.cc:2781] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:44:10.870150 8421 raft_consensus.cc:686] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 LEADER]: Becoming Leader. State: Replica: f4bf4e6e1c474e2aa7da9b99d5c1df3f, State: Running, Role: LEADER
I20260926 20:44:10.870205 8421 consensus_queue.cc:237] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } }
I20260926 20:44:10.870509 8424 tablet_replica.cc:439] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: TxnStatusTablet state changed. Reason: New leader f4bf4e6e1c474e2aa7da9b99d5c1df3f. Latest consensus state: current_term: 1 leader_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } }
I20260926 20:44:10.870587 8424 tablet_replica.cc:442] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:44:10.870803 8455 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:44:10.870890 8455 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 20:44:10.870848 7902 catalog_manager.cc:5495] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f4bf4e6e1c474e2aa7da9b99d5c1df3f (127.0.98.66). New cstate: current_term: 1 leader_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } health_report { overall_health: HEALTHY } } }
I20260926 20:44:10.915148 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.912783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46421 (local address 127.0.0.1:57648)
0926 20:44:10.912864 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:10.912877 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:10.912938 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:10.913076 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:10.913077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:10.913078 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:44:10.913130 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.913132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.913748 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.913749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.914329 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.914331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.914719 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.914720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.914784 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.914799 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:44:10.914986 (+ 187us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:44:10.914992 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:44:10.915038 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 20:44:10.915083 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 20:44:10.915140 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.912850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.0.98.66:46421)
0926 20:44:10.912919 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.912921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.912930 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:44:10.913009 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.913010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.913022 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.913072 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:44:10.913182 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.913664 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.914427 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.914644 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.914720 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.914891 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:44:10.914928 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:44:10.914980 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.915043 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.915063 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:44:10.916266 8340 raft_consensus.cc:1261] T 785978ea67c04109bb55f967ccda6019 P f5fa6c406b5548a79ad3dd8a95abb65a [term 1 FOLLOWER]: Refusing update from remote peer f4bf4e6e1c474e2aa7da9b99d5c1df3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:44:10.916294 8060 raft_consensus.cc:1261] T 785978ea67c04109bb55f967ccda6019 P 7aadd954987147dfaff804e8fd158b54 [term 1 FOLLOWER]: Refusing update from remote peer f4bf4e6e1c474e2aa7da9b99d5c1df3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:44:10.916503 8421 consensus_queue.cc:1035] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:44:10.916628 8424 consensus_queue.cc:1035] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:44:10.917086 8422 tablet_replica.cc:439] T 785978ea67c04109bb55f967ccda6019 P f5fa6c406b5548a79ad3dd8a95abb65a: TxnStatusTablet state changed. Reason: New leader f4bf4e6e1c474e2aa7da9b99d5c1df3f. Latest consensus state: current_term: 1 leader_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } }
I20260926 20:44:10.917171 8422 tablet_replica.cc:442] T 785978ea67c04109bb55f967ccda6019 P f5fa6c406b5548a79ad3dd8a95abb65a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:44:10.917122 8420 tablet_replica.cc:439] T 785978ea67c04109bb55f967ccda6019 P 7aadd954987147dfaff804e8fd158b54: TxnStatusTablet state changed. Reason: New leader f4bf4e6e1c474e2aa7da9b99d5c1df3f. Latest consensus state: current_term: 1 leader_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } }
I20260926 20:44:10.917200 8420 tablet_replica.cc:442] T 785978ea67c04109bb55f967ccda6019 P 7aadd954987147dfaff804e8fd158b54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:44:10.917853 8424 tablet_replica.cc:439] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } }
I20260926 20:44:10.917932 8424 tablet_replica.cc:442] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:44:10.917935 8427 mvcc.cc:205] Tried to move back new op lower bound from 7333705526951342080 to 7333705526764888064. Current Snapshot: MvccSnapshot[applied={T|T < 7333705526951342080}]
I20260926 20:44:10.918244 8422 tablet_replica.cc:439] T 785978ea67c04109bb55f967ccda6019 P f5fa6c406b5548a79ad3dd8a95abb65a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } }
I20260926 20:44:10.918311 8422 tablet_replica.cc:442] T 785978ea67c04109bb55f967ccda6019 P f5fa6c406b5548a79ad3dd8a95abb65a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:44:10.918363 8420 tablet_replica.cc:439] T 785978ea67c04109bb55f967ccda6019 P 7aadd954987147dfaff804e8fd158b54: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } }
I20260926 20:44:10.918437 8420 tablet_replica.cc:442] T 785978ea67c04109bb55f967ccda6019 P 7aadd954987147dfaff804e8fd158b54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:44:10.918555 8421 tablet_replica.cc:439] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } }
I20260926 20:44:10.918630 8421 tablet_replica.cc:442] T 785978ea67c04109bb55f967ccda6019 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:44:10.922667 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.920067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.0.98.67:38551)
0926 20:44:10.920108 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.920110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.920119 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:44:10.920205 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.920206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.920224 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.920290 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:44:10.920468 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.921179 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.921853 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.922151 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.922223 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.922363 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:44:10.922483 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:44:10.922556 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.922582 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.922601 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:44:10.927434 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.924961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:37393 (local address 127.0.98.66:46421)
0926 20:44:10.925006 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.925008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.925142 (+ 134us) server_negotiation.cc:407] Connection header received
0926 20:44:10.925169 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.925170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.925178 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.925243 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:44:10.925321 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.925942 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.926680 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.926960 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.927037 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.927210 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:44:10.927239 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:44:10.927298 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.927352 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.927371 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:44:10.927456 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.924890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46421 (local address 127.0.98.67:37393)
0926 20:44:10.925067 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:10.925077 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:10.925131 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:10.925247 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:10.925248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:10.925248 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:44:10.925284 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.925285 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.926019 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.926020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.926595 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.926597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.927059 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.927060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.927111 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.927123 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:44:10.927305 (+ 182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:44:10.927307 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:44:10.927342 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 20:44:10.927377 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260926 20:44:10.929724 8060 raft_consensus.cc:1261] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54 [term 1 FOLLOWER]: Refusing update from remote peer f5fa6c406b5548a79ad3dd8a95abb65a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:44:10.929780 8200 raft_consensus.cc:1261] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 FOLLOWER]: Refusing update from remote peer f5fa6c406b5548a79ad3dd8a95abb65a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:44:10.929989 8422 consensus_queue.cc:1035] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:44:10.930084 8443 consensus_queue.cc:1035] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:44:10.938249 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.935347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:38551 (local address 127.0.98.66:57251)
0926 20:44:10.935477 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:10.935488 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:10.935545 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:10.935678 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:10.935678 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:10.935679 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:44:10.935753 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.935754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.936581 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.936582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.937404 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:10.937407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.937853 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.937855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:10.937931 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.937944 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:44:10.938126 (+ 182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:44:10.938127 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:44:10.938158 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 20:44:10.938194 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
I20260926 20:44:10.938249 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.935407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:57251 (local address 127.0.98.67:38551)
0926 20:44:10.935447 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.935449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.935551 (+ 102us) server_negotiation.cc:407] Connection header received
0926 20:44:10.935598 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.935599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.935609 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.935676 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:44:10.935801 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.936491 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.937510 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.937780 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.937855 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.938030 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:44:10.938063 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:44:10.938126 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.938162 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.938181 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:44:10.947624 7903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:44:10.947768 7903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:44:10.948503 7903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79: Sending DeleteTablet for 3 replicas of tablet 4f5f87ff10fb45a3922dd48818b858d7
I20260926 20:44:10.948764 8320 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f5f87ff10fb45a3922dd48818b858d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:10 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:38228
I20260926 20:44:10.948794 8040 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f5f87ff10fb45a3922dd48818b858d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:10 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:45272
I20260926 20:44:10.948897 8438 tablet_replica.cc:330] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54: stopping tablet replica
I20260926 20:44:10.948899 8437 tablet_replica.cc:330] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a: stopping tablet replica
I20260926 20:44:10.948953 8437 raft_consensus.cc:2229] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:44:10.948957 8438 raft_consensus.cc:2229] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:44:10.948998 8438 raft_consensus.cc:2258] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:10.949048 8437 raft_consensus.cc:2258] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:10.949255 8180 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f5f87ff10fb45a3922dd48818b858d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:10 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:57622
I20260926 20:44:10.949339 8436 tablet_replica.cc:330] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: stopping tablet replica
I20260926 20:44:10.949383 8436 raft_consensus.cc:2229] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:44:10.949420 8436 raft_consensus.cc:2258] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:10.949378 8438 ts_tablet_manager.cc:1905] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:10.949760 8437 ts_tablet_manager.cc:1905] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:10.949776 8436 ts_tablet_manager.cc:1905] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:10.951267 8437 ts_tablet_manager.cc:1918] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:44:10.951333 8437 log.cc:1196] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/4f5f87ff10fb45a3922dd48818b858d7
I20260926 20:44:10.951330 8436 ts_tablet_manager.cc:1918] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:44:10.951377 8436 log.cc:1196] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/4f5f87ff10fb45a3922dd48818b858d7
I20260926 20:44:10.951567 8437 ts_tablet_manager.cc:1939] T 4f5f87ff10fb45a3922dd48818b858d7 P f5fa6c406b5548a79ad3dd8a95abb65a: Deleting consensus metadata
I20260926 20:44:10.951596 8436 ts_tablet_manager.cc:1939] T 4f5f87ff10fb45a3922dd48818b858d7 P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Deleting consensus metadata
I20260926 20:44:10.952024 7889 catalog_manager.cc:4841] TS f4bf4e6e1c474e2aa7da9b99d5c1df3f (127.0.98.66:46421): tablet 4f5f87ff10fb45a3922dd48818b858d7 (table test-table [id=cbf082c837b44a50a0b169ea353e8205]) successfully deleted
I20260926 20:44:10.954792 8438 ts_tablet_manager.cc:1918] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:44:10.954857 8438 log.cc:1196] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/wal/wals/4f5f87ff10fb45a3922dd48818b858d7
I20260926 20:44:10.955076 8438 ts_tablet_manager.cc:1939] T 4f5f87ff10fb45a3922dd48818b858d7 P 7aadd954987147dfaff804e8fd158b54: Deleting consensus metadata
I20260926 20:44:10.955382 7890 catalog_manager.cc:4841] TS 7aadd954987147dfaff804e8fd158b54 (127.0.98.65:35245): tablet 4f5f87ff10fb45a3922dd48818b858d7 (table test-table [id=cbf082c837b44a50a0b169ea353e8205]) successfully deleted
I20260926 20:44:10.955641 7890 catalog_manager.cc:4841] TS f5fa6c406b5548a79ad3dd8a95abb65a (127.0.98.67:38551): tablet 4f5f87ff10fb45a3922dd48818b858d7 (table test-table [id=cbf082c837b44a50a0b169ea353e8205]) successfully deleted
Sep 26 20:44:10 dist-test-slave-4hd3 krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:44:10 dist-test-slave-4hd3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:10 dist-test-slave-4hd3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:44:10.985975 7963 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:10.979027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42616 (local address 127.0.98.126:37799)
0926 20:44:10.979075 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.979077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.979204 (+ 127us) server_negotiation.cc:407] Connection header received
0926 20:44:10.979279 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.979280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.979291 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.979352 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:10.980139 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.980990 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.982029 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.982336 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.982416 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.985906 (+ 3490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42616: BlockingRecv error: recv got EOF from 127.0.0.1:42616 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 20:44:10 dist-test-slave-4hd3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260926 20:44:10.995491 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:10.986981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42620 (local address 127.0.98.126:37799)
0926 20:44:10.987023 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:10.987025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:10.987084 (+ 59us) server_negotiation.cc:407] Connection header received
0926 20:44:10.987129 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:10.987130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:10.987140 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:10.987204 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:10.987771 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.988535 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.989599 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:10.989896 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:10.989967 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:10.992603 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:10.992604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:10.992605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:10.992612 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:10.994300 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:10.994688 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:10.994690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:10.994690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:10.994752 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:10.994962 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:10.994963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:10.994963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:10.995092 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:10.995190 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:10.995326 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 20:44:10.995399 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:44:10.997443 7903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:44:10.997607 7903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:44:10.998932 8180 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4840c0bf304c06a78c3ddcb7f6511c (DEFAULT_TABLE table=test-table [id=8547d699428e4034a275cfe2de7e4d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:10.998940 8040 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4840c0bf304c06a78c3ddcb7f6511c (DEFAULT_TABLE table=test-table [id=8547d699428e4034a275cfe2de7e4d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:10.999064 8040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4840c0bf304c06a78c3ddcb7f6511c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:10.999070 8180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4840c0bf304c06a78c3ddcb7f6511c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:10.999841 8320 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4840c0bf304c06a78c3ddcb7f6511c (DEFAULT_TABLE table=test-table [id=8547d699428e4034a275cfe2de7e4d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:10.999979 8320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4840c0bf304c06a78c3ddcb7f6511c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:11.000535 8415 tablet_bootstrap.cc:492] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Bootstrap starting.
I20260926 20:44:11.000837 8415 tablet_bootstrap.cc:654] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:11.001174 8417 tablet_bootstrap.cc:492] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a: Bootstrap starting.
I20260926 20:44:11.001598 8417 tablet_bootstrap.cc:654] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:11.002218 8417 tablet_bootstrap.cc:492] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a: No bootstrap required, opened a new log
I20260926 20:44:11.002265 8417 ts_tablet_manager.cc:1397] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:44:11.002420 8414 tablet_bootstrap.cc:492] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54: Bootstrap starting.
I20260926 20:44:11.002413 8417 raft_consensus.cc:348] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:11.002476 8417 raft_consensus.cc:374] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:11.002497 8417 raft_consensus.cc:729] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fa6c406b5548a79ad3dd8a95abb65a, State: Initialized, Role: FOLLOWER
I20260926 20:44:11.002538 8417 consensus_queue.cc:260] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:11.002616 8417 ts_tablet_manager.cc:1428] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:11.002789 8414 tablet_bootstrap.cc:654] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:11.003552 8414 tablet_bootstrap.cc:492] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54: No bootstrap required, opened a new log
I20260926 20:44:11.003602 8414 ts_tablet_manager.cc:1397] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:11.003750 8414 raft_consensus.cc:348] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:11.003823 8414 raft_consensus.cc:374] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:11.003842 8414 raft_consensus.cc:729] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aadd954987147dfaff804e8fd158b54, State: Initialized, Role: FOLLOWER
I20260926 20:44:11.003891 8414 consensus_queue.cc:260] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:11.003974 8414 ts_tablet_manager.cc:1428] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:11.004228 8415 tablet_bootstrap.cc:492] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f: No bootstrap required, opened a new log
I20260926 20:44:11.004277 8415 ts_tablet_manager.cc:1397] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260926 20:44:11.004374 8415 raft_consensus.cc:348] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:11.004446 8415 raft_consensus.cc:374] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:11.004465 8415 raft_consensus.cc:729] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4bf4e6e1c474e2aa7da9b99d5c1df3f, State: Initialized, Role: FOLLOWER
I20260926 20:44:11.004503 8415 consensus_queue.cc:260] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:11.004580 8415 ts_tablet_manager.cc:1428] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:11.051149 8460 raft_consensus.cc:482] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:44:11.051244 8460 raft_consensus.cc:504] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:11.051542 8460 leader_election.cc:288] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7aadd954987147dfaff804e8fd158b54 (127.0.98.65:35245), f5fa6c406b5548a79ad3dd8a95abb65a (127.0.98.67:38551)
I20260926 20:44:11.051805 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4840c0bf304c06a78c3ddcb7f6511c" candidate_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" is_pre_election: true
I20260926 20:44:11.051801 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4840c0bf304c06a78c3ddcb7f6511c" candidate_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aadd954987147dfaff804e8fd158b54" is_pre_election: true
I20260926 20:44:11.051899 8059 raft_consensus.cc:2454] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4bf4e6e1c474e2aa7da9b99d5c1df3f in term 0.
I20260926 20:44:11.051936 8340 raft_consensus.cc:2454] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4bf4e6e1c474e2aa7da9b99d5c1df3f in term 0.
I20260926 20:44:11.052078 8134 leader_election.cc:302] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aadd954987147dfaff804e8fd158b54, f4bf4e6e1c474e2aa7da9b99d5c1df3f; no voters:
I20260926 20:44:11.052219 8460 raft_consensus.cc:2781] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:44:11.052258 8460 raft_consensus.cc:482] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:44:11.052284 8460 raft_consensus.cc:3037] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:11.052995 8460 raft_consensus.cc:504] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:11.053124 8460 leader_election.cc:288] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [CANDIDATE]: Term 1 election: Requested vote from peers 7aadd954987147dfaff804e8fd158b54 (127.0.98.65:35245), f5fa6c406b5548a79ad3dd8a95abb65a (127.0.98.67:38551)
I20260926 20:44:11.053277 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4840c0bf304c06a78c3ddcb7f6511c" candidate_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aadd954987147dfaff804e8fd158b54"
I20260926 20:44:11.053282 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4840c0bf304c06a78c3ddcb7f6511c" candidate_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a"
I20260926 20:44:11.053368 8059 raft_consensus.cc:3037] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:11.053367 8340 raft_consensus.cc:3037] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:11.054092 8059 raft_consensus.cc:2454] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4bf4e6e1c474e2aa7da9b99d5c1df3f in term 1.
I20260926 20:44:11.054155 8340 raft_consensus.cc:2454] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4bf4e6e1c474e2aa7da9b99d5c1df3f in term 1.
I20260926 20:44:11.054267 8134 leader_election.cc:302] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aadd954987147dfaff804e8fd158b54, f4bf4e6e1c474e2aa7da9b99d5c1df3f; no voters:
I20260926 20:44:11.054406 8460 raft_consensus.cc:2781] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:44:11.054471 8460 raft_consensus.cc:686] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 LEADER]: Becoming Leader. State: Replica: f4bf4e6e1c474e2aa7da9b99d5c1df3f, State: Running, Role: LEADER
I20260926 20:44:11.054517 8460 consensus_queue.cc:237] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:11.055114 7903 catalog_manager.cc:5495] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f4bf4e6e1c474e2aa7da9b99d5c1df3f (127.0.98.66). New cstate: current_term: 1 leader_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } health_report { overall_health: UNKNOWN } } }
I20260926 20:44:11.063058 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:11.060842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57650 (local address 127.0.98.66:46421)
0926 20:44:11.060927 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:11.060929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:11.060937 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:44:11.060958 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:11.060959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:11.060969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:11.061037 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:44:11.061147 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:11.061610 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:11.062347 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:11.062552 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:11.062627 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:11.062782 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:44:11.062813 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:44:11.062899 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:11.062963 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 20:44:11.062980 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:44:11.063736 8340 raft_consensus.cc:1261] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a [term 1 FOLLOWER]: Refusing update from remote peer f4bf4e6e1c474e2aa7da9b99d5c1df3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:44:11.063745 8059 raft_consensus.cc:1261] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54 [term 1 FOLLOWER]: Refusing update from remote peer f4bf4e6e1c474e2aa7da9b99d5c1df3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:44:11.063930 8460 consensus_queue.cc:1035] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:44:11.064004 8421 consensus_queue.cc:1035] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:44:11.066962 7902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:44:11.067078 7902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:44:11.067701 7902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79: Sending DeleteTablet for 3 replicas of tablet 8e4840c0bf304c06a78c3ddcb7f6511c
I20260926 20:44:11.067984 8180 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4840c0bf304c06a78c3ddcb7f6511c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:11 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:57622
I20260926 20:44:11.068084 8320 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4840c0bf304c06a78c3ddcb7f6511c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:11 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:38228
I20260926 20:44:11.068171 8437 tablet_replica.cc:330] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a: stopping tablet replica
I20260926 20:44:11.068187 8040 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4840c0bf304c06a78c3ddcb7f6511c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:11 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:45272
I20260926 20:44:11.068239 8437 raft_consensus.cc:2229] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:44:11.068270 8436 tablet_replica.cc:330] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f: stopping tablet replica
I20260926 20:44:11.068287 8437 raft_consensus.cc:2258] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:11.068332 8436 raft_consensus.cc:2229] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:44:11.068396 8436 raft_consensus.cc:2258] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:11.068466 8438 tablet_replica.cc:330] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54: stopping tablet replica
I20260926 20:44:11.068519 8438 raft_consensus.cc:2229] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:44:11.068562 8438 raft_consensus.cc:2258] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:11.068646 8437 ts_tablet_manager.cc:1905] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:11.068854 8438 ts_tablet_manager.cc:1905] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:11.068936 7902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:44:11.069067 7902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:44:11.070117 8180 tablet_service.cc:1467] Processing CreateTablet for tablet 79a0755475254dea8ad0d63d1c7d12fb (DEFAULT_TABLE table=test-table [id=3f5ac18f36d24ad8a8c2d1cb81dbac4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:11.070226 8180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79a0755475254dea8ad0d63d1c7d12fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:11.070333 8438 ts_tablet_manager.cc:1918] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:44:11.070389 8438 log.cc:1196] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-0/wal/wals/8e4840c0bf304c06a78c3ddcb7f6511c
I20260926 20:44:11.070596 8438 ts_tablet_manager.cc:1939] T 8e4840c0bf304c06a78c3ddcb7f6511c P 7aadd954987147dfaff804e8fd158b54: Deleting consensus metadata
I20260926 20:44:11.071461 8415 tablet_bootstrap.cc:492] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Bootstrap starting.
I20260926 20:44:11.071795 8415 tablet_bootstrap.cc:654] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:11.071851 8320 tablet_service.cc:1467] Processing CreateTablet for tablet 79a0755475254dea8ad0d63d1c7d12fb (DEFAULT_TABLE table=test-table [id=3f5ac18f36d24ad8a8c2d1cb81dbac4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:11.071990 8040 tablet_service.cc:1467] Processing CreateTablet for tablet 79a0755475254dea8ad0d63d1c7d12fb (DEFAULT_TABLE table=test-table [id=3f5ac18f36d24ad8a8c2d1cb81dbac4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:11.072081 8040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79a0755475254dea8ad0d63d1c7d12fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:11.072386 8415 tablet_bootstrap.cc:492] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f: No bootstrap required, opened a new log
I20260926 20:44:11.072436 8415 ts_tablet_manager.cc:1397] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:44:11.072531 8415 raft_consensus.cc:348] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:11.072608 8415 raft_consensus.cc:374] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:11.072629 8415 raft_consensus.cc:729] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4bf4e6e1c474e2aa7da9b99d5c1df3f, State: Initialized, Role: FOLLOWER
I20260926 20:44:11.072675 8415 consensus_queue.cc:260] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:11.072782 8415 ts_tablet_manager.cc:1428] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:11.073235 8414 tablet_bootstrap.cc:492] T 79a0755475254dea8ad0d63d1c7d12fb P 7aadd954987147dfaff804e8fd158b54: Bootstrap starting.
I20260926 20:44:11.073873 8414 tablet_bootstrap.cc:654] T 79a0755475254dea8ad0d63d1c7d12fb P 7aadd954987147dfaff804e8fd158b54: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:11.074317 8437 ts_tablet_manager.cc:1918] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:44:11.074383 8437 log.cc:1196] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/8e4840c0bf304c06a78c3ddcb7f6511c
I20260926 20:44:11.074575 8437 ts_tablet_manager.cc:1939] T 8e4840c0bf304c06a78c3ddcb7f6511c P f5fa6c406b5548a79ad3dd8a95abb65a: Deleting consensus metadata
I20260926 20:44:11.074705 8320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79a0755475254dea8ad0d63d1c7d12fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:11.074906 7890 catalog_manager.cc:4841] TS f5fa6c406b5548a79ad3dd8a95abb65a (127.0.98.67:38551): tablet 8e4840c0bf304c06a78c3ddcb7f6511c (table test-table [id=8547d699428e4034a275cfe2de7e4d30]) successfully deleted
I20260926 20:44:11.075626 8417 tablet_bootstrap.cc:492] T 79a0755475254dea8ad0d63d1c7d12fb P f5fa6c406b5548a79ad3dd8a95abb65a: Bootstrap starting.
I20260926 20:44:11.075927 7890 catalog_manager.cc:4841] TS 7aadd954987147dfaff804e8fd158b54 (127.0.98.65:35245): tablet 8e4840c0bf304c06a78c3ddcb7f6511c (table test-table [id=8547d699428e4034a275cfe2de7e4d30]) successfully deleted
I20260926 20:44:11.075959 8417 tablet_bootstrap.cc:654] T 79a0755475254dea8ad0d63d1c7d12fb P f5fa6c406b5548a79ad3dd8a95abb65a: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:11.076216 8436 ts_tablet_manager.cc:1905] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:11.076524 8414 tablet_bootstrap.cc:492] T 79a0755475254dea8ad0d63d1c7d12fb P 7aadd954987147dfaff804e8fd158b54: No bootstrap required, opened a new log
I20260926 20:44:11.076575 8414 ts_tablet_manager.cc:1397] T 79a0755475254dea8ad0d63d1c7d12fb P 7aadd954987147dfaff804e8fd158b54: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 20:44:11.076717 8414 raft_consensus.cc:348] T 79a0755475254dea8ad0d63d1c7d12fb P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:11.076786 8414 raft_consensus.cc:374] T 79a0755475254dea8ad0d63d1c7d12fb P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:11.076812 8414 raft_consensus.cc:729] T 79a0755475254dea8ad0d63d1c7d12fb P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aadd954987147dfaff804e8fd158b54, State: Initialized, Role: FOLLOWER
I20260926 20:44:11.076858 8414 consensus_queue.cc:260] T 79a0755475254dea8ad0d63d1c7d12fb P 7aadd954987147dfaff804e8fd158b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:11.076969 8414 ts_tablet_manager.cc:1428] T 79a0755475254dea8ad0d63d1c7d12fb P 7aadd954987147dfaff804e8fd158b54: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:11.077463 8460 raft_consensus.cc:482] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:44:11.077524 8460 raft_consensus.cc:504] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:11.077641 8460 leader_election.cc:288] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7aadd954987147dfaff804e8fd158b54 (127.0.98.65:35245), f5fa6c406b5548a79ad3dd8a95abb65a (127.0.98.67:38551)
I20260926 20:44:11.077811 8436 ts_tablet_manager.cc:1918] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:44:11.077875 8436 log.cc:1196] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/8e4840c0bf304c06a78c3ddcb7f6511c
I20260926 20:44:11.078091 8436 ts_tablet_manager.cc:1939] T 8e4840c0bf304c06a78c3ddcb7f6511c P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Deleting consensus metadata
I20260926 20:44:11.078389 7889 catalog_manager.cc:4841] TS f4bf4e6e1c474e2aa7da9b99d5c1df3f (127.0.98.66:46421): tablet 8e4840c0bf304c06a78c3ddcb7f6511c (table test-table [id=8547d699428e4034a275cfe2de7e4d30]) successfully deleted
I20260926 20:44:11.078366 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79a0755475254dea8ad0d63d1c7d12fb" candidate_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" is_pre_election: true
W20260926 20:44:11.078639 8134 leader_election.cc:341] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f5fa6c406b5548a79ad3dd8a95abb65a (127.0.98.67:38551): Illegal state: must be running to vote when last-logged opid is not known
I20260926 20:44:11.080483 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79a0755475254dea8ad0d63d1c7d12fb" candidate_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aadd954987147dfaff804e8fd158b54" is_pre_election: true
I20260926 20:44:11.080578 8059 raft_consensus.cc:2454] T 79a0755475254dea8ad0d63d1c7d12fb P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4bf4e6e1c474e2aa7da9b99d5c1df3f in term 0.
I20260926 20:44:11.080705 8134 leader_election.cc:302] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7aadd954987147dfaff804e8fd158b54, f4bf4e6e1c474e2aa7da9b99d5c1df3f; no voters: f5fa6c406b5548a79ad3dd8a95abb65a
I20260926 20:44:11.080822 8460 raft_consensus.cc:2781] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:44:11.080855 8460 raft_consensus.cc:482] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:44:11.080890 8460 raft_consensus.cc:3037] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:11.081398 8460 raft_consensus.cc:504] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:11.081530 8460 leader_election.cc:288] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [CANDIDATE]: Term 1 election: Requested vote from peers 7aadd954987147dfaff804e8fd158b54 (127.0.98.65:35245), f5fa6c406b5548a79ad3dd8a95abb65a (127.0.98.67:38551)
I20260926 20:44:11.081670 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79a0755475254dea8ad0d63d1c7d12fb" candidate_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a"
W20260926 20:44:11.081863 8134 leader_election.cc:341] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f5fa6c406b5548a79ad3dd8a95abb65a (127.0.98.67:38551): Illegal state: must be running to vote when last-logged opid is not known
I20260926 20:44:11.081944 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79a0755475254dea8ad0d63d1c7d12fb" candidate_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aadd954987147dfaff804e8fd158b54"
I20260926 20:44:11.082008 8059 raft_consensus.cc:3037] T 79a0755475254dea8ad0d63d1c7d12fb P 7aadd954987147dfaff804e8fd158b54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:11.082595 8417 tablet_bootstrap.cc:492] T 79a0755475254dea8ad0d63d1c7d12fb P f5fa6c406b5548a79ad3dd8a95abb65a: No bootstrap required, opened a new log
I20260926 20:44:11.082638 8417 ts_tablet_manager.cc:1397] T 79a0755475254dea8ad0d63d1c7d12fb P f5fa6c406b5548a79ad3dd8a95abb65a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260926 20:44:11.082647 8059 raft_consensus.cc:2454] T 79a0755475254dea8ad0d63d1c7d12fb P 7aadd954987147dfaff804e8fd158b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4bf4e6e1c474e2aa7da9b99d5c1df3f in term 1.
I20260926 20:44:11.082731 8417 raft_consensus.cc:348] T 79a0755475254dea8ad0d63d1c7d12fb P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:11.082792 8417 raft_consensus.cc:374] T 79a0755475254dea8ad0d63d1c7d12fb P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:11.082810 8417 raft_consensus.cc:729] T 79a0755475254dea8ad0d63d1c7d12fb P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fa6c406b5548a79ad3dd8a95abb65a, State: Initialized, Role: FOLLOWER
I20260926 20:44:11.082803 8134 leader_election.cc:302] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7aadd954987147dfaff804e8fd158b54, f4bf4e6e1c474e2aa7da9b99d5c1df3f; no voters: f5fa6c406b5548a79ad3dd8a95abb65a
I20260926 20:44:11.082849 8417 consensus_queue.cc:260] T 79a0755475254dea8ad0d63d1c7d12fb P f5fa6c406b5548a79ad3dd8a95abb65a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:11.082899 8460 raft_consensus.cc:2781] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:44:11.082926 8417 ts_tablet_manager.cc:1428] T 79a0755475254dea8ad0d63d1c7d12fb P f5fa6c406b5548a79ad3dd8a95abb65a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:11.082937 8460 raft_consensus.cc:686] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 LEADER]: Becoming Leader. State: Replica: f4bf4e6e1c474e2aa7da9b99d5c1df3f, State: Running, Role: LEADER
I20260926 20:44:11.082971 8460 consensus_queue.cc:237] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } }
I20260926 20:44:11.083494 7902 catalog_manager.cc:5495] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f4bf4e6e1c474e2aa7da9b99d5c1df3f (127.0.98.66). New cstate: current_term: 1 leader_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4bf4e6e1c474e2aa7da9b99d5c1df3f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 } health_report { overall_health: UNKNOWN } } }
I20260926 20:44:11.094148 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:11.092214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:40143 (local address 127.0.98.66:46421)
0926 20:44:11.092287 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:11.092289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:11.092298 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:44:11.092320 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:11.092320 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:11.092330 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:11.092391 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:44:11.092490 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:11.092936 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:11.093613 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:11.093782 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:11.093834 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:11.093952 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:44:11.093978 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:44:11.094016 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:11.094067 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 20:44:11.094081 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:44:11.094198 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:11.092094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46421 (local address 127.0.98.66:40143)
0926 20:44:11.092165 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:11.092177 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:11.092230 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:11.092406 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:11.092407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:11.092408 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:44:11.092447 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:11.092448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:11.092999 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:11.093000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:11.093541 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:11.093543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:11.093849 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:11.093850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:11.093886 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:11.093891 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:44:11.094031 (+ 140us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:44:11.094032 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:44:11.094066 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 20:44:11.094099 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 20:44:11.096069 8059 raft_consensus.cc:1261] T 79a0755475254dea8ad0d63d1c7d12fb P 7aadd954987147dfaff804e8fd158b54 [term 1 FOLLOWER]: Refusing update from remote peer f4bf4e6e1c474e2aa7da9b99d5c1df3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:44:11.096079 8340 raft_consensus.cc:3037] T 79a0755475254dea8ad0d63d1c7d12fb P f5fa6c406b5548a79ad3dd8a95abb65a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:11.096722 8340 raft_consensus.cc:1261] T 79a0755475254dea8ad0d63d1c7d12fb P f5fa6c406b5548a79ad3dd8a95abb65a [term 1 FOLLOWER]: Refusing update from remote peer f4bf4e6e1c474e2aa7da9b99d5c1df3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:44:11.096918 8421 consensus_queue.cc:1035] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5fa6c406b5548a79ad3dd8a95abb65a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:44:11.097023 8424 consensus_queue.cc:1035] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aadd954987147dfaff804e8fd158b54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:44:11.105867 7901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:44:11.105984 7901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:44:11.106613 7901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3786db80b7ba4acfb9fbb81abbb40a79: Sending DeleteTablet for 3 replicas of tablet 79a0755475254dea8ad0d63d1c7d12fb
I20260926 20:44:11.106835 8040 tablet_service.cc:1514] Processing DeleteTablet for tablet 79a0755475254dea8ad0d63d1c7d12fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:11 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:45272
I20260926 20:44:11.106856 8180 tablet_service.cc:1514] Processing DeleteTablet for tablet 79a0755475254dea8ad0d63d1c7d12fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:11 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:57622
I20260926 20:44:11.106875 8320 tablet_service.cc:1514] Processing DeleteTablet for tablet 79a0755475254dea8ad0d63d1c7d12fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:11 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:38228
I20260926 20:44:11.106931 8438 tablet_replica.cc:330] T 79a0755475254dea8ad0d63d1c7d12fb P 7aadd954987147dfaff804e8fd158b54: stopping tablet replica
I20260926 20:44:11.106961 8436 tablet_replica.cc:330] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f: stopping tablet replica
I20260926 20:44:11.106988 8438 raft_consensus.cc:2229] T 79a0755475254dea8ad0d63d1c7d12fb P 7aadd954987147dfaff804e8fd158b54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:44:11.107025 8436 raft_consensus.cc:2229] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:44:11.107028 8438 raft_consensus.cc:2258] T 79a0755475254dea8ad0d63d1c7d12fb P 7aadd954987147dfaff804e8fd158b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:11.107084 8436 raft_consensus.cc:2258] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:11.107100 8437 tablet_replica.cc:330] T 79a0755475254dea8ad0d63d1c7d12fb P f5fa6c406b5548a79ad3dd8a95abb65a: stopping tablet replica
I20260926 20:44:11.107157 8437 raft_consensus.cc:2229] T 79a0755475254dea8ad0d63d1c7d12fb P f5fa6c406b5548a79ad3dd8a95abb65a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:44:11.107208 8437 raft_consensus.cc:2258] T 79a0755475254dea8ad0d63d1c7d12fb P f5fa6c406b5548a79ad3dd8a95abb65a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:11.107410 8436 ts_tablet_manager.cc:1905] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:11.107642 8438 ts_tablet_manager.cc:1905] T 79a0755475254dea8ad0d63d1c7d12fb P 7aadd954987147dfaff804e8fd158b54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:11.108857 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 7975
I20260926 20:44:11.109701 8436 ts_tablet_manager.cc:1918] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:44:11.109778 8436 log.cc:1196] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/79a0755475254dea8ad0d63d1c7d12fb
I20260926 20:44:11.109979 8436 ts_tablet_manager.cc:1939] T 79a0755475254dea8ad0d63d1c7d12fb P f4bf4e6e1c474e2aa7da9b99d5c1df3f: Deleting consensus metadata
I20260926 20:44:11.110285 7889 catalog_manager.cc:4841] TS f4bf4e6e1c474e2aa7da9b99d5c1df3f (127.0.98.66:46421): tablet 79a0755475254dea8ad0d63d1c7d12fb (table test-table [id=3f5ac18f36d24ad8a8c2d1cb81dbac4c]) successfully deleted
I20260926 20:44:11.111889 8437 ts_tablet_manager.cc:1905] T 79a0755475254dea8ad0d63d1c7d12fb P f5fa6c406b5548a79ad3dd8a95abb65a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:11.113178 8437 ts_tablet_manager.cc:1918] T 79a0755475254dea8ad0d63d1c7d12fb P f5fa6c406b5548a79ad3dd8a95abb65a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:44:11.113229 8437 log.cc:1196] T 79a0755475254dea8ad0d63d1c7d12fb P f5fa6c406b5548a79ad3dd8a95abb65a: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/79a0755475254dea8ad0d63d1c7d12fb
I20260926 20:44:11.113461 8437 ts_tablet_manager.cc:1939] T 79a0755475254dea8ad0d63d1c7d12fb P f5fa6c406b5548a79ad3dd8a95abb65a: Deleting consensus metadata
I20260926 20:44:11.113766 7890 catalog_manager.cc:4841] TS f5fa6c406b5548a79ad3dd8a95abb65a (127.0.98.67:38551): tablet 79a0755475254dea8ad0d63d1c7d12fb (table test-table [id=3f5ac18f36d24ad8a8c2d1cb81dbac4c]) successfully deleted
W20260926 20:44:11.115090 7890 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:35245 (error 108)
I20260926 20:44:11.115569 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:11.115420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:35245 (local address 127.0.0.1:45282)
0926 20:44:11.115506 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:11.115538 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:35245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":43,"threads_started":1}
I20260926 20:44:11.115679 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 8115
W20260926 20:44:11.115694 7890 catalog_manager.cc:4584] TS 7aadd954987147dfaff804e8fd158b54 (127.0.98.65:35245): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79a0755475254dea8ad0d63d1c7d12fb: Network error: Client connection negotiation failed: client connection to 127.0.98.65:35245: connect: Connection refused (error 111)
I20260926 20:44:11.121639 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 8255
I20260926 20:44:11.127007 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 7871
2026-09-26T20:44:11Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2516 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:44:11 dist-test-slave-4hd3 krb5kdc[8502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:44:11 dist-test-slave-4hd3 krb5kdc[8502](info): set up 2 sockets
Sep 26 20:44:11 dist-test-slave-4hd3 krb5kdc[8502](info): commencing operation
krb5kdc: starting...
W20260926 20:44:12.164314 393 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:44:12 dist-test-slave-4hd3 krb5kdc[8502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:44:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:44:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:44:12.283483 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38017
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:38017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:12.363137 8515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:12.363335 8515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:12.363356 8515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:12.364454 8515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:44:12.364490 8515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:12.364502 8515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:12.364521 8515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:44:12.364533 8515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:44:12.365931 8515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:38017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:12.366122 8515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:12.366281 8515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:12.371177 8523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:12.371348 8521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:12.371346 8515 server_base.cc:1034] running on GCE node
W20260926 20:44:12.371177 8520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:12.371773 8515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:12.372020 8515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:12.373157 8515 hybrid_clock.cc:648] HybridClock initialized: now 1790455452373149 us; error 32 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T20:44:12Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1253 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:44:12 dist-test-slave-4hd3 krb5kdc[8528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:44:12 dist-test-slave-4hd3 krb5kdc[8528](info): set up 2 sockets
Sep 26 20:44:12 dist-test-slave-4hd3 krb5kdc[8528](info): commencing operation
krb5kdc: starting...
W20260926 20:44:13.408267 393 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:44:13 dist-test-slave-4hd3 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:44:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:44:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:44:13.522384 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:35517
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:35517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:13.601059 8541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:13.601311 8541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:13.601347 8541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:13.602479 8541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:44:13.602558 8541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:13.602586 8541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:13.602618 8541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:44:13.602640 8541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:44:13.604151 8541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:35517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:35517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:13.604419 8541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:13.604650 8541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:13.609341 8547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:13.609400 8546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:13.609530 8549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:13.609822 8541 server_base.cc:1034] running on GCE node
I20260926 20:44:13.610139 8541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:13.610474 8541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:13.611645 8541 hybrid_clock.cc:648] HybridClock initialized: now 1790455453611623 us; error 39 us; skew 500 ppm
Sep 26 20:44:13 dist-test-slave-4hd3 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:13.618948 8541 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:44:13.619406 8541 webserver.cc:460] Webserver started at http://127.0.98.126:46273/ using document root <none> and password file <none>
I20260926 20:44:13.619585 8541 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:13.619625 8541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:13.619763 8541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:13.620823 8541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "3bc75729ec5e48e3b80daa158a8abf23"
format_stamp: "Formatted at 2026-09-26 20:44:13 on dist-test-slave-4hd3"
server_key: "84cc450326253601840d45899b62656e"
server_key_iv: "f5c0c6074d44c298f5b6cd6cb52d3057"
server_key_version: "encryptionkey@0"
I20260926 20:44:13.621132 8541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "3bc75729ec5e48e3b80daa158a8abf23"
format_stamp: "Formatted at 2026-09-26 20:44:13 on dist-test-slave-4hd3"
server_key: "84cc450326253601840d45899b62656e"
server_key_iv: "f5c0c6074d44c298f5b6cd6cb52d3057"
server_key_version: "encryptionkey@0"
I20260926 20:44:13.622375 8541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:44:13.623059 8556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:13.623231 8541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:44:13.623315 8541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "3bc75729ec5e48e3b80daa158a8abf23"
format_stamp: "Formatted at 2026-09-26 20:44:13 on dist-test-slave-4hd3"
server_key: "84cc450326253601840d45899b62656e"
server_key_iv: "f5c0c6074d44c298f5b6cd6cb52d3057"
server_key_version: "encryptionkey@0"
I20260926 20:44:13.623368 8541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:13.638962 8541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:13.641324 8541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:13.641441 8541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:13.645223 8541 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:35517
I20260926 20:44:13.645277 8608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:35517 every 8 connection(s)
I20260926 20:44:13.645745 8541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:44:13.646471 8609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:13.646745 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 8541
I20260926 20:44:13.646849 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:44:13.647060 393 external_mini_cluster.cc:1428] Setting key aee66f290c0f1c2bae276fa3b1484f44
I20260926 20:44:13.649549 8609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23: Bootstrap starting.
I20260926 20:44:13.650172 8609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:13.650472 8609 log.cc:824] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23: Log is configured to *not* fsync() on all Append() calls
I20260926 20:44:13.651185 8609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23: No bootstrap required, opened a new log
Sep 26 20:44:13 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:44:13.653756 8609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc75729ec5e48e3b80daa158a8abf23" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35517 } }
I20260926 20:44:13.653908 8609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:13.653961 8609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bc75729ec5e48e3b80daa158a8abf23, State: Initialized, Role: FOLLOWER
I20260926 20:44:13.654091 8609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc75729ec5e48e3b80daa158a8abf23" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35517 } }
I20260926 20:44:13.654181 8609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:44:13.654254 8609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:44:13.654309 8609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:13.655050 8609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc75729ec5e48e3b80daa158a8abf23" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35517 } }
I20260926 20:44:13.655166 8609 leader_election.cc:302] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bc75729ec5e48e3b80daa158a8abf23; no voters:
I20260926 20:44:13.655346 8609 leader_election.cc:288] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:44:13.655378 8614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:44:13.655591 8609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:44:13.655584 8614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [term 1 LEADER]: Becoming Leader. State: Replica: 3bc75729ec5e48e3b80daa158a8abf23, State: Running, Role: LEADER
I20260926 20:44:13.656262 8614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc75729ec5e48e3b80daa158a8abf23" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35517 } }
I20260926 20:44:13.656957 8616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bc75729ec5e48e3b80daa158a8abf23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc75729ec5e48e3b80daa158a8abf23" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35517 } } }
I20260926 20:44:13.657104 8615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bc75729ec5e48e3b80daa158a8abf23. Latest consensus state: current_term: 1 leader_uuid: "3bc75729ec5e48e3b80daa158a8abf23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc75729ec5e48e3b80daa158a8abf23" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35517 } } }
I20260926 20:44:13.657567 8616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:13.657578 8615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [sys.catalog]: This master's current role is: LEADER
W20260926 20:44:13.657716 8628 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:44:13.657779 8628 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:44:13.657843 8632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:44:13.658016 8632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:44:13.658156 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:13.647602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.0.98.126:35517)
0926 20:44:13.647784 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:13.647800 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:13.647825 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:44:13.648237 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:13.648242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:13.648287 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:13.648401 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:13.649168 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:13.650083 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:13.651085 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:13.651527 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:13.651618 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:13.654152 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:13.654156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:13.654156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:13.654167 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:13.656083 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:13.656472 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:13.656476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:13.656476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:13.656596 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:13.656845 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:13.656846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:13.656846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:13.657176 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:13.657322 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:13.657901 (+ 579us) server_negotiation.cc:299] Negotiation successful
0926 20:44:13.658022 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
I20260926 20:44:13.659497 8632 catalog_manager.cc:1347] Generated new cluster ID: 21853c5c9a1d410ba395d9f8457cb694
I20260926 20:44:13.659564 8632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:44:13.675266 8632 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:44:13.675868 8632 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:44:13.690568 8632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23: Generated new TSK 0
I20260926 20:44:13.690758 8632 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260926 20:44:13.716747 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35517
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:13.796440 8636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:13.796636 8636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:13.796658 8636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:13.797827 8636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:13.797868 8636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:13.797894 8636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:44:13.799309 8636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:13.799532 8636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:13.799813 8636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:13.804716 8642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:13.804744 8636 server_base.cc:1034] running on GCE node
W20260926 20:44:13.804744 8644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:13.804716 8641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:13.805157 8636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:13.805341 8636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:13.806481 8636 hybrid_clock.cc:648] HybridClock initialized: now 1790455453806461 us; error 37 us; skew 500 ppm
Sep 26 20:44:13 dist-test-slave-4hd3 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:13.808920 8636 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:44:13.809281 8636 webserver.cc:460] Webserver started at http://127.0.98.65:44453/ using document root <none> and password file <none>
I20260926 20:44:13.809460 8636 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:13.809499 8636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:13.809583 8636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:13.810550 8636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "bce92589980d471482de76a4756a96c8"
format_stamp: "Formatted at 2026-09-26 20:44:13 on dist-test-slave-4hd3"
server_key: "af4c23685ff5faaf71a2997fc56614aa"
server_key_iv: "7a296783d67a0dc7e769820ce3cb302a"
server_key_version: "encryptionkey@0"
I20260926 20:44:13.810933 8636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "bce92589980d471482de76a4756a96c8"
format_stamp: "Formatted at 2026-09-26 20:44:13 on dist-test-slave-4hd3"
server_key: "af4c23685ff5faaf71a2997fc56614aa"
server_key_iv: "7a296783d67a0dc7e769820ce3cb302a"
server_key_version: "encryptionkey@0"
I20260926 20:44:13.812232 8636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:44:13.812922 8651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:13.813088 8636 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:13.813189 8636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "bce92589980d471482de76a4756a96c8"
format_stamp: "Formatted at 2026-09-26 20:44:13 on dist-test-slave-4hd3"
server_key: "af4c23685ff5faaf71a2997fc56614aa"
server_key_iv: "7a296783d67a0dc7e769820ce3cb302a"
server_key_version: "encryptionkey@0"
I20260926 20:44:13.813264 8636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:13.831185 8636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:13.833570 8636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:13.833745 8636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:13.833982 8636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:13.834290 8636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:13.834342 8636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:13.834376 8636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:13.834403 8636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:13.840596 8636 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:34051
I20260926 20:44:13.840637 8764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:34051 every 8 connection(s)
I20260926 20:44:13.840969 8636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:44:13.841364 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 8636
I20260926 20:44:13.841486 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:44:13.841668 393 external_mini_cluster.cc:1428] Setting key 8566094275dfd0855b88b355ef4c3e80
Sep 26 20:44:13 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260926 20:44:13.852223 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:13.841823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:55567 (local address 127.0.98.126:35517)
0926 20:44:13.841876 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:13.841878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:13.842166 (+ 288us) server_negotiation.cc:407] Connection header received
0926 20:44:13.843579 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:13.843580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:13.843590 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:13.843645 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:13.844419 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:13.845215 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:13.846292 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:13.846648 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:13.846732 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:13.849283 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:13.849285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:13.849285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:13.849294 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:13.850803 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:13.851198 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:13.851200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:13.851200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:13.851260 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:13.851620 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:13.851620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:13.851620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:13.851762 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:13.851825 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:13.852091 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 20:44:13.852157 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 20:44:13.852555 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:13.841786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.65:55567)
0926 20:44:13.842004 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:13.842058 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:13.843462 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:13.843747 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:13.843751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:13.844119 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:13.844317 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:13.844323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:13.845306 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:13.845307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:13.846192 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:13.846194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:13.846737 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:13.846738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:13.846834 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:13.847516 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:13.847525 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:13.849184 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:13.850878 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:13.850883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:13.850885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:13.851107 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:13.851425 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:13.851426 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:13.851426 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:13.851547 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:13.851827 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:13.851831 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:13.851948 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 20:44:13.852067 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 20:44:13.852320 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260926 20:44:13.853102 8765 heartbeater.cc:347] Connected to a master server at 127.0.98.126:35517
I20260926 20:44:13.853215 8765 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:13.853397 8765 heartbeater.cc:511] Master 127.0.98.126:35517 requested a full tablet report, sending...
I20260926 20:44:13.853953 8573 ts_manager.cc:194] Registered new tserver with Master: bce92589980d471482de76a4756a96c8 (127.0.98.65:34051)
I20260926 20:44:13.854727 8573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:55567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260926 20:44:13.866945 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35517
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:13.946404 8770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:13.946609 8770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:13.946635 8770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:13.947778 8770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:13.947813 8770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:13.947839 8770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:44:13.949291 8770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:13.949481 8770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:13.949658 8770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:13.954697 8776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:13.954692 8778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:13.954692 8775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:13.954887 8770 server_base.cc:1034] running on GCE node
I20260926 20:44:13.955242 8770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:13.955544 8770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:13.956718 8770 hybrid_clock.cc:648] HybridClock initialized: now 1790455453956702 us; error 34 us; skew 500 ppm
Sep 26 20:44:13 dist-test-slave-4hd3 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:13.959524 8770 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:44:13.960103 8770 webserver.cc:460] Webserver started at http://127.0.98.66:44051/ using document root <none> and password file <none>
I20260926 20:44:13.960331 8770 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:13.960402 8770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:13.960561 8770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:13.961632 8770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "c738e71584e84c74a619a16868c0b8b8"
format_stamp: "Formatted at 2026-09-26 20:44:13 on dist-test-slave-4hd3"
server_key: "f413460c5410dbca4d442b607dd608cc"
server_key_iv: "0d9ae1f70b1b939fae9679b27556256c"
server_key_version: "encryptionkey@0"
I20260926 20:44:13.962005 8770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "c738e71584e84c74a619a16868c0b8b8"
format_stamp: "Formatted at 2026-09-26 20:44:13 on dist-test-slave-4hd3"
server_key: "f413460c5410dbca4d442b607dd608cc"
server_key_iv: "0d9ae1f70b1b939fae9679b27556256c"
server_key_version: "encryptionkey@0"
I20260926 20:44:13.963366 8770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:44:13.964282 8785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:13.964514 8770 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:44:13.964620 8770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "c738e71584e84c74a619a16868c0b8b8"
format_stamp: "Formatted at 2026-09-26 20:44:13 on dist-test-slave-4hd3"
server_key: "f413460c5410dbca4d442b607dd608cc"
server_key_iv: "0d9ae1f70b1b939fae9679b27556256c"
server_key_version: "encryptionkey@0"
I20260926 20:44:13.964701 8770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:13.972672 8770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:13.974987 8770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:13.975144 8770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:13.975368 8770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:13.975675 8770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:13.975749 8770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:13.975793 8770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:13.975826 8770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:13.981913 8770 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:45825
I20260926 20:44:13.981952 8898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:45825 every 8 connection(s)
I20260926 20:44:13.982278 8770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:44:13 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:44:13.991449 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 8770
I20260926 20:44:13.991468 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:13.982990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:38519 (local address 127.0.98.126:35517)
0926 20:44:13.983046 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:13.983047 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:13.983244 (+ 197us) server_negotiation.cc:407] Connection header received
0926 20:44:13.984289 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:13.984290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:13.984302 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:13.984359 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:13.984990 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:13.985485 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:13.986469 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:13.986665 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:13.986769 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:13.988824 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:13.988826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:13.988826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:13.988834 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:13.990116 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:13.990518 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:13.990520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:13.990520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:13.990577 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:13.990825 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:13.990826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:13.990826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:13.990936 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:13.991011 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:13.991300 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 20:44:13.991382 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 20:44:13.991600 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:44:13.991673 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:13.982969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.66:38519)
0926 20:44:13.983131 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:13.983163 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:13.984179 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:13.984456 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:13.984459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:13.984712 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:13.984896 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:13.984900 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:13.985576 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:13.985577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:13.986380 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:13.986382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:13.986775 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:13.986776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:13.986894 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:13.987360 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:13.987375 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:13.988731 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:13.990210 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:13.990216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:13.990218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:13.990430 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:13.990644 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:13.990645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:13.990645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:13.990753 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:13.991002 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:13.991008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:13.991157 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 20:44:13.991262 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 20:44:13.991413 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260926 20:44:13.991820 393 external_mini_cluster.cc:1428] Setting key de396c267e3af1e0676e014a57fc22e6
I20260926 20:44:13.992195 8899 heartbeater.cc:347] Connected to a master server at 127.0.98.126:35517
I20260926 20:44:13.992301 8899 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:13.992497 8899 heartbeater.cc:511] Master 127.0.98.126:35517 requested a full tablet report, sending...
I20260926 20:44:13.992869 8573 ts_manager.cc:194] Registered new tserver with Master: c738e71584e84c74a619a16868c0b8b8 (127.0.98.66:45825)
I20260926 20:44:13.993240 8573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:38519
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260926 20:44:14.014313 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35517
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:14.094060 8904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:14.094247 8904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:14.094270 8904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:14.095340 8904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:14.095374 8904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:14.095400 8904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:44:14.096881 8904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:14.097069 8904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:14.097226 8904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:14.101934 8912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:14.102032 8910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:14.102138 8909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:14.102150 8904 server_base.cc:1034] running on GCE node
I20260926 20:44:14.102404 8904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:14.102646 8904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:14.103771 8904 hybrid_clock.cc:648] HybridClock initialized: now 1790455454103761 us; error 39 us; skew 500 ppm
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:14.106235 8904 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:44:14.106642 8904 webserver.cc:460] Webserver started at http://127.0.98.67:35057/ using document root <none> and password file <none>
I20260926 20:44:14.106868 8904 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:14.106935 8904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:14.107085 8904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:14.108140 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "88dba92ad39842c1902206fb328e8e84"
format_stamp: "Formatted at 2026-09-26 20:44:14 on dist-test-slave-4hd3"
server_key: "1d7847b110fdd54e0dbdc40215bda0f3"
server_key_iv: "20cb2756df2ea6dfb3a44ad7f49b6db4"
server_key_version: "encryptionkey@0"
I20260926 20:44:14.108505 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "88dba92ad39842c1902206fb328e8e84"
format_stamp: "Formatted at 2026-09-26 20:44:14 on dist-test-slave-4hd3"
server_key: "1d7847b110fdd54e0dbdc40215bda0f3"
server_key_iv: "20cb2756df2ea6dfb3a44ad7f49b6db4"
server_key_version: "encryptionkey@0"
I20260926 20:44:14.109766 8904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:44:14.110469 8919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:14.110631 8904 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:44:14.110731 8904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "88dba92ad39842c1902206fb328e8e84"
format_stamp: "Formatted at 2026-09-26 20:44:14 on dist-test-slave-4hd3"
server_key: "1d7847b110fdd54e0dbdc40215bda0f3"
server_key_iv: "20cb2756df2ea6dfb3a44ad7f49b6db4"
server_key_version: "encryptionkey@0"
I20260926 20:44:14.110801 8904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:14.125388 8904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:14.127627 8904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:14.127803 8904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:14.128024 8904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:14.128324 8904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:14.128376 8904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:14.128417 8904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:14.128444 8904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:14.134294 8904 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:36531
I20260926 20:44:14.134346 9032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:36531 every 8 connection(s)
I20260926 20:44:14.134641 8904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:44:14.138911 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 8904
I20260926 20:44:14.139051 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:44:14.139262 393 external_mini_cluster.cc:1428] Setting key 37526d9b3ad7ff642797ee283f978ad9
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:44:14.144075 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:14.135389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:50619 (local address 127.0.98.126:35517)
0926 20:44:14.135434 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:14.135436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:14.135657 (+ 221us) server_negotiation.cc:407] Connection header received
0926 20:44:14.136689 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:14.136691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:14.136704 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:14.136747 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:14.137366 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.137879 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.139000 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.139281 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.139369 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.141546 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:14.141547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:14.141548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:14.141554 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:14.142847 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:14.143212 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:14.143213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:14.143213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:14.143264 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:14.143491 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:14.143492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:14.143492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:14.143601 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:14.143651 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:14.143935 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 20:44:14.144012 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:44:14.144330 9035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:14.135367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.67:50619)
0926 20:44:14.135541 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:14.135575 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:14.136566 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:14.136829 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:14.136833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:14.137092 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:14.137274 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.137278 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.137977 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.137978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.138877 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.138880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.139437 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.139438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.139548 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.140145 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:14.140157 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:14.141463 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:14.142917 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:14.142923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:14.142924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:14.143146 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:14.143321 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:14.143322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:14.143322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:14.143425 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:14.143653 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:14.143657 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:14.143836 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 20:44:14.143922 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 20:44:14.144044 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260926 20:44:14.144764 9033 heartbeater.cc:347] Connected to a master server at 127.0.98.126:35517
I20260926 20:44:14.144870 9033 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:14.145057 9033 heartbeater.cc:511] Master 127.0.98.126:35517 requested a full tablet report, sending...
I20260926 20:44:14.145570 8573 ts_manager.cc:194] Registered new tserver with Master: 88dba92ad39842c1902206fb328e8e84 (127.0.98.67:36531)
I20260926 20:44:14.145972 8573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:50619
I20260926 20:44:14.151126 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/oryx_127.0.98.65.keytab.
Entry for principal oryx/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/oryx_127.0.98.65.keytab.
WARNING: no policy specified for oryx/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/oryx_127.0.98.66.keytab.
Entry for principal oryx/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/oryx_127.0.98.66.keytab.
WARNING: no policy specified for oryx/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/oryx_127.0.98.67.keytab.
Entry for principal oryx/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/oryx_127.0.98.67.keytab.
I20260926 20:44:14.222059 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 8636
I20260926 20:44:14.226745 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 8770
I20260926 20:44:14.230880 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 8904
I20260926 20:44:14.235059 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 8541
I20260926 20:44:14.240662 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:35517
--webserver_interface=127.0.98.126
--webserver_port=46273
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:35517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:14.323837 9042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:14.324033 9042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:14.324055 9042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:14.325187 9042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:44:14.325228 9042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:14.325242 9042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:14.325255 9042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:44:14.325306 9042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:44:14.326828 9042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:35517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:35517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=46273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:14.327070 9042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:14.327312 9042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:14.332255 9047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:14.332260 9050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:14.332268 9048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:14.332573 9042 server_base.cc:1034] running on GCE node
I20260926 20:44:14.332839 9042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:14.333144 9042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:14.334319 9042 hybrid_clock.cc:648] HybridClock initialized: now 1790455454334290 us; error 44 us; skew 500 ppm
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:14.336862 9042 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:44:14.337338 9042 webserver.cc:460] Webserver started at http://127.0.98.126:46273/ using document root <none> and password file <none>
I20260926 20:44:14.337570 9042 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:14.337626 9042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:14.339043 9042 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:14.339824 9057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:14.340008 9042 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:14.340108 9042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "3bc75729ec5e48e3b80daa158a8abf23"
format_stamp: "Formatted at 2026-09-26 20:44:13 on dist-test-slave-4hd3"
server_key: "84cc450326253601840d45899b62656e"
server_key_iv: "f5c0c6074d44c298f5b6cd6cb52d3057"
server_key_version: "encryptionkey@0"
I20260926 20:44:14.340463 9042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:14.359005 9042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:14.361238 9042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:14.361413 9042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:14.365664 9042 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:35517
I20260926 20:44:14.365713 9109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:35517 every 8 connection(s)
I20260926 20:44:14.366251 9042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:44:14.367484 9110 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:44:14.368155 9110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23: Bootstrap starting.
I20260926 20:44:14.375248 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 9042
I20260926 20:44:14.375537 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:34051
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:35517
--webserver_port=44453
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.65
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/oryx_127.0.98.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:44:14.376160 9110 log.cc:824] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23: Log is configured to *not* fsync() on all Append() calls
I20260926 20:44:14.378227 9110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:44:14.378520 9110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23: Bootstrap complete.
I20260926 20:44:14.380825 9110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc75729ec5e48e3b80daa158a8abf23" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35517 } }
I20260926 20:44:14.381042 9110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bc75729ec5e48e3b80daa158a8abf23, State: Initialized, Role: FOLLOWER
I20260926 20:44:14.381211 9110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc75729ec5e48e3b80daa158a8abf23" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35517 } }
I20260926 20:44:14.381278 9110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:44:14.381337 9110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:44:14.381384 9110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:44:14.382354 9110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc75729ec5e48e3b80daa158a8abf23" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35517 } }
I20260926 20:44:14.382447 9110 leader_election.cc:302] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bc75729ec5e48e3b80daa158a8abf23; no voters:
I20260926 20:44:14.382630 9110 leader_election.cc:288] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:44:14.382663 9114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:44:14.382762 9114 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [term 2 LEADER]: Becoming Leader. State: Replica: 3bc75729ec5e48e3b80daa158a8abf23, State: Running, Role: LEADER
I20260926 20:44:14.382875 9110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:44:14.383049 9114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc75729ec5e48e3b80daa158a8abf23" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35517 } }
I20260926 20:44:14.383373 9116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bc75729ec5e48e3b80daa158a8abf23. Latest consensus state: current_term: 2 leader_uuid: "3bc75729ec5e48e3b80daa158a8abf23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc75729ec5e48e3b80daa158a8abf23" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35517 } } }
I20260926 20:44:14.383374 9115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3bc75729ec5e48e3b80daa158a8abf23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc75729ec5e48e3b80daa158a8abf23" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35517 } } }
I20260926 20:44:14.383508 9116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:14.383519 9115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23 [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:14.383756 9123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:44:14.384289 9123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:44:14.384472 9123 catalog_manager.cc:1259] Loaded cluster ID: 21853c5c9a1d410ba395d9f8457cb694
I20260926 20:44:14.384681 9123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:44:14.385854 9123 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:44:14.386030 9123 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3bc75729ec5e48e3b80daa158a8abf23: Loaded TSK: 0
I20260926 20:44:14.386243 9123 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:44:14.457077 9113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:14.457319 9113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:14.457358 9113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:14.458590 9113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:14.458659 9113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:14.458706 9113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:44:14.460561 9113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:34051
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=44453
--tserver_master_addrs=127.0.98.126:35517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:14.460829 9113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:14.461066 9113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:14.466967 9137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:14.467051 9139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:14.466974 9136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:14.467085 9113 server_base.cc:1034] running on GCE node
I20260926 20:44:14.467391 9113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:14.467674 9113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:14.468871 9113 hybrid_clock.cc:648] HybridClock initialized: now 1790455454468848 us; error 44 us; skew 500 ppm
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455454, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:14.471843 9113 init.cc:377] Logged in from keytab as oryx/127.0.98.65@KRBTEST.COM (short username oryx)
I20260926 20:44:14.472285 9113 webserver.cc:460] Webserver started at http://127.0.98.65:44453/ using document root <none> and password file <none>
I20260926 20:44:14.472530 9113 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:14.472612 9113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:14.474004 9113 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:14.474614 9146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:14.474817 9113 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:44:14.474900 9113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "bce92589980d471482de76a4756a96c8"
format_stamp: "Formatted at 2026-09-26 20:44:13 on dist-test-slave-4hd3"
server_key: "af4c23685ff5faaf71a2997fc56614aa"
server_key_iv: "7a296783d67a0dc7e769820ce3cb302a"
server_key_version: "encryptionkey@0"
I20260926 20:44:14.475327 9113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:14.497859 9113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:14.500370 9113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:14.500510 9113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:14.500747 9113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:14.501109 9113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:14.501151 9113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:14.501180 9113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:14.501197 9113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:14.508185 9113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:34051
I20260926 20:44:14.508221 9259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:34051 every 8 connection(s)
I20260926 20:44:14.508659 9113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:44:14.510883 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 9113
I20260926 20:44:14.511166 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:45825
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:35517
--webserver_port=44051
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.66
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/oryx_127.0.98.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:14.519230 9262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:14.509381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.65:34613)
0926 20:44:14.509607 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:14.509653 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:14.510279 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:14.510736 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:14.510740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:14.511481 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:14.511835 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.511842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.513182 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.513183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.514404 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.514407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.515205 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.515207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.515365 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.516033 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:14.516044 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:14.518993 (+ 2949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":108,"threads_started":1}
W20260926 20:44:14.519517 9260 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:35517 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.126:35517: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
W20260926 20:44:14.519544 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:14.509498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:34613 (local address 127.0.98.126:35517)
0926 20:44:14.509683 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:14.509690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:14.509742 (+ 52us) server_negotiation.cc:407] Connection header received
0926 20:44:14.510478 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:14.510484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:14.510531 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:14.510641 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:14.512013 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.513062 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.514499 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.515109 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.515214 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.519137 (+ 3923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:34613: BlockingRecv error: recv got EOF from 127.0.98.65:34613 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":69,"threads_started":1}
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:14.524775 9262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:14.519865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.65:40505)
0926 20:44:14.519910 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:14.519920 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:14.519973 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:14.520132 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:14.520133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:14.520293 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:14.520346 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.520348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.521210 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.521211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.522124 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.522127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.522474 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.522475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.522541 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.522926 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:14.522931 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:14.524696 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 20:44:14.524827 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:14.519806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:40505 (local address 127.0.98.126:35517)
0926 20:44:14.519865 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:14.519867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:14.519974 (+ 107us) server_negotiation.cc:407] Connection header received
0926 20:44:14.520036 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:14.520037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:14.520049 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:14.520108 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:14.520426 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.521119 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.522211 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.522390 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.522450 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.524766 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:40505: BlockingRecv error: recv got EOF from 127.0.98.65:40505 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:14.529425 9262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:14.525119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.65:58817)
0926 20:44:14.525158 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:14.525168 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:14.525202 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:14.525335 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:14.525336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:14.525444 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:14.525468 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.525469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.526317 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.526318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.526785 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.526786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.527261 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.527262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.527306 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.527764 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:14.527775 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:14.529362 (+ 1587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 20:44:14.529523 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:14.525160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:58817 (local address 127.0.98.126:35517)
0926 20:44:14.525218 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:14.525220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:14.525229 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:44:14.525252 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:14.525253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:14.525264 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:14.525325 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:14.525531 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.526245 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.526857 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.527168 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.527233 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.529436 (+ 2203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:58817: BlockingRecv error: recv got EOF from 127.0.98.65:58817 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 20:44:14.529724 9260 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:44:14.596333 9264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:14.596566 9264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:14.596603 9264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:14.597770 9264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:14.597836 9264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:14.597880 9264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:44:14.599303 9264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:45825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=44051
--tserver_master_addrs=127.0.98.126:35517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:14.599566 9264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:14.599839 9264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:14.604843 9272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:14.604897 9270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:14.604851 9269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:14.605072 9264 server_base.cc:1034] running on GCE node
I20260926 20:44:14.605248 9264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:14.605432 9264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:14.606580 9264 hybrid_clock.cc:648] HybridClock initialized: now 1790455454606557 us; error 36 us; skew 500 ppm
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455454, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:14.609058 9264 init.cc:377] Logged in from keytab as oryx/127.0.98.66@KRBTEST.COM (short username oryx)
I20260926 20:44:14.609477 9264 webserver.cc:460] Webserver started at http://127.0.98.66:44051/ using document root <none> and password file <none>
I20260926 20:44:14.609690 9264 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:14.609745 9264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:14.611127 9264 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:44:14.611693 9279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:14.611909 9264 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:44:14.612007 9264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "c738e71584e84c74a619a16868c0b8b8"
format_stamp: "Formatted at 2026-09-26 20:44:13 on dist-test-slave-4hd3"
server_key: "f413460c5410dbca4d442b607dd608cc"
server_key_iv: "0d9ae1f70b1b939fae9679b27556256c"
server_key_version: "encryptionkey@0"
I20260926 20:44:14.612346 9264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:14.624665 9264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:14.626828 9264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:14.627005 9264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:14.627254 9264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:14.627556 9264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:14.627610 9264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:14.627650 9264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:14.627679 9264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:14.633435 9264 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:45825
I20260926 20:44:14.633522 9392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:45825 every 8 connection(s)
I20260926 20:44:14.633865 9264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:44:14.636669 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 9264
I20260926 20:44:14.636960 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:36531
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:35517
--webserver_port=35057
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.67
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/oryx_127.0.98.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:14.642427 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:14.634606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.66:41043)
0926 20:44:14.634819 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:14.634864 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:14.635388 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:14.635679 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:14.635683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:14.636257 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:14.636491 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.636498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.637353 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.637354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.638588 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.638591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.638997 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.638998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.639163 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.639763 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:14.639774 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:14.642275 (+ 2501us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
W20260926 20:44:14.642498 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:14.634649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:41043 (local address 127.0.98.126:35517)
0926 20:44:14.634771 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:14.634774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:14.634999 (+ 225us) server_negotiation.cc:407] Connection header received
0926 20:44:14.635522 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:14.635524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:14.635540 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:14.635597 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:14.636605 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.637262 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.638683 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.638897 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.638988 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.642401 (+ 3413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:41043: BlockingRecv error: recv got EOF from 127.0.98.66:41043 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 20:44:14.642819 9393 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:35517 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.126:35517: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:14.647902 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:14.643075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.66:33257)
0926 20:44:14.643179 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:14.643188 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:14.643228 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:14.643377 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:14.643377 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:14.643498 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:14.643543 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.643544 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.644589 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.644591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.645091 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.645093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.645598 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.645599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.645652 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.646147 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:14.646151 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:14.647852 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20260926 20:44:14.648012 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:14.643133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:33257 (local address 127.0.98.126:35517)
0926 20:44:14.643202 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:14.643205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:14.643220 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:44:14.643298 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:14.643299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:14.643311 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:14.643375 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:14.643614 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.644478 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.645177 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.645519 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.645595 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.647951 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:33257: BlockingRecv error: recv got EOF from 127.0.98.66:33257 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:14.652698 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:14.648319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.66:42725)
0926 20:44:14.648414 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:14.648425 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:14.648479 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:14.648596 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:14.648597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:14.648735 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:14.648766 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.648767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.649372 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.649372 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.650173 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.650175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.650482 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.650483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.650535 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.650886 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:14.650891 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:14.652625 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":81}
W20260926 20:44:14.652776 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:14.648375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:42725 (local address 127.0.98.126:35517)
0926 20:44:14.648417 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:14.648419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:14.648459 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:44:14.648537 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:14.648537 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:14.648544 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:14.648584 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:14.648824 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.649304 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.650249 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.650414 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.650460 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.652704 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:42725: BlockingRecv error: recv got EOF from 127.0.98.66:42725 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 20:44:14.652809 9393 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:44:14.719502 9396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:14.719707 9396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:14.719749 9396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:14.720925 9396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:14.720973 9396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:14.721024 9396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:44:14.722438 9396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:36531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=35057
--tserver_master_addrs=127.0.98.126:35517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:14.722678 9396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:14.722890 9396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:14.727648 9404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:14.727735 9401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:14.727916 9396 server_base.cc:1034] running on GCE node
W20260926 20:44:14.727648 9402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:14.728206 9396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:14.728440 9396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:14.729615 9396 hybrid_clock.cc:648] HybridClock initialized: now 1790455454729602 us; error 27 us; skew 500 ppm
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455454, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:14.732422 9396 init.cc:377] Logged in from keytab as oryx/127.0.98.67@KRBTEST.COM (short username oryx)
I20260926 20:44:14.732894 9396 webserver.cc:460] Webserver started at http://127.0.98.67:35057/ using document root <none> and password file <none>
I20260926 20:44:14.733141 9396 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:14.733198 9396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:14.734601 9396 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:14.735220 9411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:14.735414 9396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:44:14.735518 9396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "88dba92ad39842c1902206fb328e8e84"
format_stamp: "Formatted at 2026-09-26 20:44:14 on dist-test-slave-4hd3"
server_key: "1d7847b110fdd54e0dbdc40215bda0f3"
server_key_iv: "20cb2756df2ea6dfb3a44ad7f49b6db4"
server_key_version: "encryptionkey@0"
I20260926 20:44:14.735925 9396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:14.751116 9396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:14.753067 9396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:14.753229 9396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:14.753465 9396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:14.753763 9396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:14.753816 9396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:14.753851 9396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:14.753878 9396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:14.759531 9396 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:36531
I20260926 20:44:14.759668 9524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:36531 every 8 connection(s)
I20260926 20:44:14.759976 9396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:44:14.762181 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 9396
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:44:14.767861 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:14.760723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:43669 (local address 127.0.98.126:35517)
0926 20:44:14.760772 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:14.760774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:14.760959 (+ 185us) server_negotiation.cc:407] Connection header received
0926 20:44:14.761470 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:14.761471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:14.761481 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:14.761533 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:14.762932 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.763492 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.764554 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.764753 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.764837 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.767802 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:43669: BlockingRecv error: recv got EOF from 127.0.98.67:43669 (error 108)
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":25}
I20260926 20:44:14.767880 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:14.760693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.67:43669)
0926 20:44:14.760858 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:14.760891 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:14.761358 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:14.761589 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:14.761592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:14.762280 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:14.762838 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.762844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.763586 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.763587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.764440 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.764442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.764846 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.764847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.764978 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.765504 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:14.765515 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:14.767687 (+ 2172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
W20260926 20:44:14.768198 9525 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:35517 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.126:35517: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
I20260926 20:44:14.769837 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:14.762702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55846 (local address 127.0.98.126:35517)
0926 20:44:14.762818 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:14.762820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:14.762828 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:44:14.762850 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:14.762851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:14.762861 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:14.762941 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:14.763537 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.764099 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.764860 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.765066 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.765138 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.766413 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:14.766417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:14.766418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:14.766427 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:14.768431 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:14.768840 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:14.768844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:14.768844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:14.768969 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:14.769168 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:14.769169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:14.769169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:14.769367 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:14.769470 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:14.769636 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 20:44:14.769744 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:14.772966 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:14.768425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.67:56133)
0926 20:44:14.768531 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:14.768539 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:14.768583 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:14.768693 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:14.768693 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:14.768819 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:14.768868 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.768870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.769630 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.769631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.770363 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.770365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.770751 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.770752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.770816 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.771152 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:14.771156 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:14.772873 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94}
W20260926 20:44:14.773094 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:14.768488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:56133 (local address 127.0.98.126:35517)
0926 20:44:14.768525 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:14.768526 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:14.768575 (+ 49us) server_negotiation.cc:407] Connection header received
0926 20:44:14.768620 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:14.768621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:14.768630 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:14.768691 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:14.768942 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.769559 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.770448 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.770681 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.770754 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.772987 (+ 2233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:56133: BlockingRecv error: recv got EOF from 127.0.98.67:56133 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:14 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:14.776872 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:14.773238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.67:36627)
0926 20:44:14.773267 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:14.773282 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:14.773318 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:14.773420 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:14.773420 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:14.773519 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:14.773545 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.773546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.774143 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.774144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.774610 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:14.774611 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.774911 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.774911 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:14.774946 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.775276 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:14.775280 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:14.776830 (+ 1550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":14}
W20260926 20:44:14.777014 9525 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:44:14.777020 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:14.773287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:36627 (local address 127.0.98.126:35517)
0926 20:44:14.773320 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:14.773321 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:14.773355 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:44:14.773371 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:14.773372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:14.773379 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:14.773415 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:14.773632 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.774063 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.774681 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:14.774845 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:14.774894 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:14.776958 (+ 2064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:36627: BlockingRecv error: recv got EOF from 127.0.98.67:36627 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 26 20:44:15 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:15 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:15.539463 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:15.530148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.65:48031)
0926 20:44:15.530302 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:15.530312 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:15.530361 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:15.530550 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:15.530552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:15.530715 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:15.530766 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:15.530767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:15.531667 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:15.531668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:15.532287 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:15.532289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:15.532622 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:15.532623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:15.532661 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:15.537158 (+ 4497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:15.537163 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:15.539390 (+ 2227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
W20260926 20:44:15.539644 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:15.530264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:48031 (local address 127.0.98.126:35517)
0926 20:44:15.530391 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:15.530393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:15.530402 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:44:15.530425 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:15.530426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:15.530436 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:15.530504 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:15.530842 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:15.531561 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:15.532360 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:15.532556 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:15.532611 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:15.539552 (+ 6941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:48031: BlockingRecv error: recv got EOF from 127.0.98.65:48031 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":80,"threads_started":1}
Sep 26 20:44:15 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:15 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:15.657780 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:15.653196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.66:56517)
0926 20:44:15.653329 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:15.653340 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:15.653392 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:15.653532 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:15.653533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:15.653695 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:15.653743 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:15.653745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:15.654395 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:15.654395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:15.654942 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:15.654943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:15.655316 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:15.655317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:15.655361 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:15.655949 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:15.655954 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:15.657708 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
W20260926 20:44:15.657918 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:15.653295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:56517 (local address 127.0.98.126:35517)
0926 20:44:15.653363 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:15.653365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:15.653397 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:44:15.653458 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:15.653459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:15.653469 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:15.653512 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:15.653810 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:15.654327 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:15.655052 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:15.655241 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:15.655299 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:15.657836 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:56517: BlockingRecv error: recv got EOF from 127.0.98.66:56517 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 20:44:15 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:15 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:15.782277 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:15.777436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.67:39047)
0926 20:44:15.777567 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:15.777585 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:15.777635 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:15.777770 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:15.777770 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:15.777928 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:15.777976 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:15.777978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:15.778652 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:15.778653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:15.779312 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:15.779315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:15.779639 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:15.779640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:15.779677 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:15.780313 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:15.780318 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:15.782212 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":88,"threads_started":1}
W20260926 20:44:15.782392 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:15.777576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:39047 (local address 127.0.98.126:35517)
0926 20:44:15.777623 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:15.777625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:15.777642 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:44:15.777684 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:15.777684 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:15.777696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:15.777755 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:15.778063 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:15.778572 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:15.779388 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:15.779575 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:15.779628 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:15.782294 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:39047: BlockingRecv error: recv got EOF from 127.0.98.67:39047 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 20:44:16 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:16 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:16.544788 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:16.539938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.65:52833)
0926 20:44:16.540137 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:16.540148 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:16.540222 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:16.540346 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:16.540347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:16.540499 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:16.540546 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:16.540565 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:16.541299 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:16.541300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:16.542065 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:16.542069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:16.542404 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:16.542405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:16.542444 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:16.542913 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:16.542920 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:16.544729 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":117,"threads_started":1}
W20260926 20:44:16.544940 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:16.540044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:52833 (local address 127.0.98.126:35517)
0926 20:44:16.540146 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:16.540148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:16.540226 (+ 78us) server_negotiation.cc:407] Connection header received
0926 20:44:16.540266 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:16.540266 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:16.540277 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:16.540335 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:16.540653 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:16.541209 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:16.542151 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:16.542342 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:16.542396 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:16.544849 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:52833: BlockingRecv error: recv got EOF from 127.0.98.65:52833 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":69,"threads_started":1}
Sep 26 20:44:16 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:16 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:16.662875 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:16.658304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.66:41709)
0926 20:44:16.658451 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:16.658469 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:16.658527 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:16.658668 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:16.658669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:16.658829 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:16.658878 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:16.658880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:16.659547 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:16.659548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:16.660127 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:16.660129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:16.660476 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:16.660476 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:16.660516 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:16.661087 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:16.661093 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:16.662813 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
W20260926 20:44:16.663023 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:16.658416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:41709 (local address 127.0.98.126:35517)
0926 20:44:16.658487 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:16.658489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:16.658521 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:44:16.658590 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:16.658591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:16.658602 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:16.658646 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:16.658941 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:16.659468 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:16.660211 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:16.660403 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:16.660469 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:16.662943 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:41709: BlockingRecv error: recv got EOF from 127.0.98.66:41709 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 20:44:16 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:16 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:16.787628 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:16.782727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.67:39227)
0926 20:44:16.782905 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:16.782918 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:16.782978 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:16.783180 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:16.783181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:16.783339 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:16.783387 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:16.783388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:16.784308 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:16.784309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:16.784839 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:16.784841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:16.785226 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:16.785227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:16.785290 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:16.785760 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:16.785764 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:16.787565 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":111,"threads_started":1}
W20260926 20:44:16.787778 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:16.782869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:39227 (local address 127.0.98.126:35517)
0926 20:44:16.782941 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:16.782958 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:16.782968 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:44:16.783086 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:16.783087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:16.783108 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:16.783159 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:16.783463 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:16.784236 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:16.784923 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:16.785157 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:16.785216 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:16.787669 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:39227: BlockingRecv error: recv got EOF from 127.0.98.67:39227 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 20:44:17 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:17 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:17.549757 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:17.545247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.65:49483)
0926 20:44:17.545363 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:17.545380 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:17.545436 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:17.545643 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:17.545644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:17.545801 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:17.545849 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:17.545851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:17.546547 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:17.546548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:17.547089 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:17.547090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:17.547431 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:17.547432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:17.547472 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:17.547930 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:17.547934 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:17.549693 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":67,"threads_started":1}
W20260926 20:44:17.549901 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:17.545283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:49483 (local address 127.0.98.126:35517)
0926 20:44:17.545402 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:17.545404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:17.545418 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:44:17.545551 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:17.545552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:17.545564 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:17.545627 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:17.545915 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:17.546473 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:17.547167 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:17.547364 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:17.547422 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:17.549798 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:49483: BlockingRecv error: recv got EOF from 127.0.98.65:49483 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
Sep 26 20:44:17 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:17 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:17.668372 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:17.663383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.66:50275)
0926 20:44:17.663681 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:17.663692 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:17.663780 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:17.663945 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:17.663946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:17.664218 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:17.664273 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:17.664276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:17.664952 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:17.664954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:17.665496 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:17.665497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:17.665864 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:17.665864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:17.665919 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:17.666436 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:17.666442 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:17.668280 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":187,"threads_started":1}
W20260926 20:44:17.668411 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:17.663616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:50275 (local address 127.0.98.126:35517)
0926 20:44:17.663687 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:17.663689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:17.663780 (+ 91us) server_negotiation.cc:407] Connection header received
0926 20:44:17.663843 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:17.663844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:17.663854 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:17.663924 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:17.664361 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:17.664867 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:17.665585 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:17.665780 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:17.665865 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:17.668351 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:50275: BlockingRecv error: recv got EOF from 127.0.98.66:50275 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 20:44:17 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:17 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:17.792910 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:17.788149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.67:33673)
0926 20:44:17.788304 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:17.788315 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:17.788371 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:17.788523 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:17.788524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:17.788686 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:17.788732 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:17.788734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:17.789425 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:17.789426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:17.789954 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:17.789956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:17.790365 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:17.790366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:17.790410 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:17.790952 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:17.790957 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:17.792854 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
W20260926 20:44:17.793156 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:17.788270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:33673 (local address 127.0.98.126:35517)
0926 20:44:17.788319 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:17.788325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:17.788360 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:44:17.788428 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:17.788428 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:17.788448 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:17.788492 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:17.788800 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:17.789323 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:17.790105 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:17.790291 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:17.790348 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:17.793074 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:33673: BlockingRecv error: recv got EOF from 127.0.98.67:33673 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 20:44:18 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:18 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:18.555143 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:18.550326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.65:35035)
0926 20:44:18.550471 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:18.550486 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:18.550542 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:18.550704 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:18.550704 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:18.550867 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:18.550914 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:18.550916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:18.551601 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:18.551602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:18.552488 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:18.552491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:18.552846 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:18.552846 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:18.552885 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:18.553333 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:18.553338 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:18.555086 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
W20260926 20:44:18.555305 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:18.550402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:35035 (local address 127.0.98.126:35517)
0926 20:44:18.550557 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:18.550560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:18.550570 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:44:18.550601 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:18.550601 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:18.550611 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:18.550687 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:18.551029 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:18.551530 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:18.552568 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:18.552776 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:18.552839 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:18.555197 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:35035: BlockingRecv error: recv got EOF from 127.0.98.65:35035 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":109,"threads_started":1}
Sep 26 20:44:18 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:18 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:18.677815 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:18.668885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.66:44223)
0926 20:44:18.669089 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:18.669101 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:18.669174 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:18.669311 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:18.669312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:18.669510 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:18.669556 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:18.669558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:18.670285 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:18.670287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:18.670834 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:18.670835 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:18.671218 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:18.671219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:18.671276 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:18.675914 (+ 4638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:18.675920 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:18.677721 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
W20260926 20:44:18.677865 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:18.668995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:44223 (local address 127.0.98.126:35517)
0926 20:44:18.669070 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:18.669073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:18.669155 (+ 82us) server_negotiation.cc:407] Connection header received
0926 20:44:18.669233 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:18.669234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:18.669246 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:18.669293 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:18.669642 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:18.670195 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:18.670920 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:18.671132 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:18.671217 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:18.677802 (+ 6585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:44223: BlockingRecv error: recv got EOF from 127.0.98.66:44223 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 20:44:18 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:18 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:18.798074 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:18.793429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.67:49087)
0926 20:44:18.793593 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:18.793604 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:18.793664 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:18.793770 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:18.793770 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:18.793933 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:18.793988 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:18.793990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:18.794637 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:18.794639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:18.795191 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:18.795193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:18.795516 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:18.795517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:18.795562 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:18.796128 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:18.796133 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:18.798014 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":113,"threads_started":1}
W20260926 20:44:18.798151 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:18.793493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:49087 (local address 127.0.98.126:35517)
0926 20:44:18.793563 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:18.793565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:18.793666 (+ 101us) server_negotiation.cc:407] Connection header received
0926 20:44:18.793704 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:18.793705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:18.793717 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:18.793766 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:18.794056 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:18.794551 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:18.795269 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:18.795443 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:18.795507 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:18.798096 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:49087: BlockingRecv error: recv got EOF from 127.0.98.67:49087 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 20:44:19 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:19 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:19.560776 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:19.555754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.65:42543)
0926 20:44:19.555886 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:19.555897 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:19.556000 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:19.556211 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:19.556212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:19.556380 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:19.556429 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:19.556431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:19.557126 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:19.557127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:19.557649 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:19.557651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:19.558062 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:19.558063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:19.558103 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:19.558613 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:19.558617 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:19.560717 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
W20260926 20:44:19.560891 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:19.555801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:42543 (local address 127.0.98.126:35517)
0926 20:44:19.555927 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:19.555929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:19.555937 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:44:19.556112 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:19.556113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:19.556125 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:19.556197 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:19.556502 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:19.557029 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:19.557751 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:19.557978 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:19.558062 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:19.560818 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:42543: BlockingRecv error: recv got EOF from 127.0.98.65:42543 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
Sep 26 20:44:19 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:19 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:19.682924 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:19.678236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.66:37823)
0926 20:44:19.678365 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:19.678375 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:19.678420 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:19.678562 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:19.678563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:19.678723 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:19.678769 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:19.678770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:19.679489 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:19.679490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:19.680090 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:19.680093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:19.680430 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:19.680431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:19.680481 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:19.680996 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:19.681001 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:19.682856 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
W20260926 20:44:19.682991 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:19.678394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:37823 (local address 127.0.98.126:35517)
0926 20:44:19.678457 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:19.678459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:19.678467 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:44:19.678487 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:19.678488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:19.678497 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:19.678551 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:19.678864 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:19.679415 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:19.680154 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:19.680337 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:19.680392 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:19.682942 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:37823: BlockingRecv error: recv got EOF from 127.0.98.66:37823 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 20:44:19 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:19 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:19.803165 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:19.798629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.67:44437)
0926 20:44:19.798789 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:19.798805 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:19.798865 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:19.798987 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:19.798988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:19.799149 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:19.799196 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:19.799198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:19.799864 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:19.799865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:19.800391 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:19.800393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:19.800718 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:19.800719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:19.800761 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:19.801218 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:19.801223 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:19.803110 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":76,"threads_started":1}
W20260926 20:44:19.803285 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:19.798734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:44437 (local address 127.0.98.126:35517)
0926 20:44:19.798785 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:19.798787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:19.798851 (+ 64us) server_negotiation.cc:407] Connection header received
0926 20:44:19.798916 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:19.798917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:19.798928 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:19.798968 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:19.799259 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:19.799774 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:19.800465 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:19.800654 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:19.800715 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:19.803205 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:44437: BlockingRecv error: recv got EOF from 127.0.98.67:44437 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 26 20:44:20 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:20 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:20.566061 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:20.561322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.65:45599)
0926 20:44:20.561452 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:20.561467 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:20.561523 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:20.561694 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:20.561695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:20.561857 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:20.561905 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:20.561907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:20.562625 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:20.562626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:20.563167 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:20.563169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:20.563545 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:20.563545 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:20.563590 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:20.564076 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:20.564081 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:20.565950 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
W20260926 20:44:20.566113 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:20.561382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:45599 (local address 127.0.98.126:35517)
0926 20:44:20.561511 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:20.561513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:20.561524 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:44:20.561572 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:20.561573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:20.561583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:20.561654 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:20.562032 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:20.562547 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:20.563257 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:20.563451 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:20.563516 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:20.566042 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:45599: BlockingRecv error: recv got EOF from 127.0.98.65:45599 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
Sep 26 20:44:20 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:20 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:20.688263 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:20.683454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.66:39871)
0926 20:44:20.683611 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:20.683623 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:20.683689 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:20.683889 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:20.683890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:20.684129 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:20.684176 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:20.684178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:20.684814 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:20.684815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:20.685377 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:20.685378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:20.685733 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:20.685734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:20.685793 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:20.686264 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:20.686269 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:20.688208 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
W20260926 20:44:20.688372 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:20.683529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:39871 (local address 127.0.98.126:35517)
0926 20:44:20.683617 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:20.683620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:20.683786 (+ 166us) server_negotiation.cc:407] Connection header received
0926 20:44:20.683817 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:20.683818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:20.683831 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:20.683875 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:20.684239 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:20.684747 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:20.685455 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:20.685639 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:20.685697 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:20.688298 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:39871: BlockingRecv error: recv got EOF from 127.0.98.66:39871 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 20:44:20 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:20 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:20.808393 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:20.803668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.67:57291)
0926 20:44:20.803814 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:20.803826 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:20.803891 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:20.804026 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:20.804026 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:20.804182 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:20.804230 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:20.804232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:20.804845 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:20.804846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:20.805363 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:20.805364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:20.805697 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:20.805698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:20.805751 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:20.806268 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:20.806273 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:20.808326 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
W20260926 20:44:20.808506 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:20.803765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:57291 (local address 127.0.98.126:35517)
0926 20:44:20.803826 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:20.803828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:20.803889 (+ 61us) server_negotiation.cc:407] Connection header received
0926 20:44:20.803948 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:20.803948 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:20.803960 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:20.804003 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:20.804290 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:20.804778 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:20.805436 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:20.805633 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:20.805687 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:20.808412 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:57291: BlockingRecv error: recv got EOF from 127.0.98.67:57291 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 20:44:21 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:21 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:21.570938 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:21.566490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.65:39703)
0926 20:44:21.566633 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:21.566644 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:21.566699 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:21.566948 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:21.566949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:21.567103 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:21.567172 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:21.567174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:21.567828 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:21.567829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:21.568356 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:21.568358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:21.568699 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:21.568700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:21.568738 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:21.569230 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:21.569235 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:21.570882 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
W20260926 20:44:21.571164 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:21.566699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:39703 (local address 127.0.98.126:35517)
0926 20:44:21.566828 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:21.566829 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:21.566837 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:44:21.566858 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:21.566858 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:21.566869 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:21.566933 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:21.567237 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:21.567762 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:21.568432 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:21.568637 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:21.568692 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:21.571092 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:39703: BlockingRecv error: recv got EOF from 127.0.98.65:39703 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
Sep 26 20:44:21 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:21 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:21.693259 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:21.688735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.66:46367)
0926 20:44:21.688893 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:21.688904 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:21.688959 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:21.689090 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:21.689091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:21.689257 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:21.689301 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:21.689303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:21.689934 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:21.689935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:21.690466 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:21.690468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:21.690792 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:21.690792 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:21.690833 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:21.691338 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:21.691343 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:21.693203 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
W20260926 20:44:21.693356 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:21.688826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:46367 (local address 127.0.98.126:35517)
0926 20:44:21.688893 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:21.688895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:21.688958 (+ 63us) server_negotiation.cc:407] Connection header received
0926 20:44:21.689018 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:21.689018 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:21.689028 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:21.689070 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:21.689366 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:21.689860 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:21.690538 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:21.690731 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:21.690782 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:21.693288 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:46367: BlockingRecv error: recv got EOF from 127.0.98.66:46367 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 20:44:21 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:21 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:21.813444 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:21.808917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.67:50951)
0926 20:44:21.809086 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:21.809097 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:21.809165 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:21.809296 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:21.809297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:21.809472 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:21.809519 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:21.809521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:21.810210 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:21.810211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:21.810741 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:21.810743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:21.811182 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:21.811182 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:21.811244 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:21.811699 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:21.811703 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:21.813349 (+ 1646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
W20260926 20:44:21.813501 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:21.808980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:50951 (local address 127.0.98.126:35517)
0926 20:44:21.809063 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:21.809066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:21.809162 (+ 96us) server_negotiation.cc:407] Connection header received
0926 20:44:21.809223 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:21.809224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:21.809238 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:21.809291 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:21.809616 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:21.810125 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:21.810825 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:21.811095 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:21.811178 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:21.813427 (+ 2249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:50951: BlockingRecv error: recv got EOF from 127.0.98.67:50951 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 20:44:22 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:22 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:22.576038 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:22.571397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.65:52997)
0926 20:44:22.571554 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:22.571567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:22.571658 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:22.571840 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:22.571842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:22.572033 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:22.572082 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:22.572084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:22.572746 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:22.572747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:22.573304 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:22.573305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:22.573619 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:22.573620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:22.573659 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:22.574178 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:22.574183 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:22.575969 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
W20260926 20:44:22.576109 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:22.571538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:52997 (local address 127.0.98.126:35517)
0926 20:44:22.571660 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:22.571662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:22.571690 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:44:22.571728 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:22.571729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:22.571740 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:22.571791 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:22.572158 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:22.572657 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:22.573376 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:22.573560 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:22.573609 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:22.576056 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:52997: BlockingRecv error: recv got EOF from 127.0.98.65:52997 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
Sep 26 20:44:22 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:22 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:22.698374 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:22.693757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.66:36523)
0926 20:44:22.693926 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:22.693936 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:22.693997 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:22.694127 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:22.694128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:22.694284 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:22.694332 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:22.694333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:22.695134 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:22.695135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:22.695657 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:22.695659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:22.696061 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:22.696062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:22.696105 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:22.696603 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:22.696608 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:22.698292 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
W20260926 20:44:22.698493 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:22.693806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:36523 (local address 127.0.98.126:35517)
0926 20:44:22.693864 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:22.693867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:22.693992 (+ 125us) server_negotiation.cc:407] Connection header received
0926 20:44:22.694048 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:22.694049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:22.694059 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:22.694110 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:22.694414 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:22.695056 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:22.695752 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:22.695942 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:22.696048 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:22.698417 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:36523: BlockingRecv error: recv got EOF from 127.0.98.66:36523 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 20:44:22 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:22 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:22.818689 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:22.813909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.67:37767)
0926 20:44:22.814077 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:22.814089 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:22.814152 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:22.814319 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:22.814320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:22.814473 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:22.814521 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:22.814522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:22.815142 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:22.815142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:22.815701 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:22.815702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:22.816109 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:22.816110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:22.816176 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:22.816653 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:22.816657 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:22.818632 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":95,"threads_started":1}
W20260926 20:44:22.818775 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:22.813961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:37767 (local address 127.0.98.126:35517)
0926 20:44:22.814009 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:22.814011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:22.814148 (+ 137us) server_negotiation.cc:407] Connection header received
0926 20:44:22.814194 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:22.814195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:22.814206 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:22.814295 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:22.814596 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:22.815078 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:22.815809 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:22.815987 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:22.816074 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:22.818715 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:37767: BlockingRecv error: recv got EOF from 127.0.98.67:37767 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 20:44:23 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:23 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:23.581542 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:23.576574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.65:50109)
0926 20:44:23.576830 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:23.576844 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:23.576942 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:23.577111 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:23.577112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:23.577278 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:23.577327 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:23.577329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:23.578118 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:23.578119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:23.578713 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:23.578716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:23.579115 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:23.579116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:23.579166 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:23.579634 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:23.579653 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:23.581479 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":82,"threads_started":1}
W20260926 20:44:23.581657 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:23.576777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:50109 (local address 127.0.98.126:35517)
0926 20:44:23.576920 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:23.576921 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:23.576931 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:44:23.577020 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:23.577020 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:23.577031 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:23.577092 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:23.577387 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:23.578039 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:23.578810 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:23.579020 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:23.579083 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:23.581564 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:50109: BlockingRecv error: recv got EOF from 127.0.98.65:50109 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
Sep 26 20:44:23 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:23 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:23.708639 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:23.698884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.66:57315)
0926 20:44:23.699052 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:23.699065 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:23.699133 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:23.699297 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:23.699298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:23.699471 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:23.699515 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:23.699517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:23.700257 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:23.700258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:23.700806 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:23.700807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:23.701189 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:23.701190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:23.701250 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:23.706358 (+ 5108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:23.706364 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:23.708532 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
W20260926 20:44:23.708680 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:23.698986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:57315 (local address 127.0.98.126:35517)
0926 20:44:23.699102 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:23.699104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:23.699114 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:44:23.699209 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:23.699210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:23.699223 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:23.699292 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:23.699625 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:23.700182 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:23.700885 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:23.701108 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:23.701191 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:23.708617 (+ 7426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:57315: BlockingRecv error: recv got EOF from 127.0.98.66:57315 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 20:44:23 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:23 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:23.823930 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:23.819131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.67:49041)
0926 20:44:23.819331 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:23.819342 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:23.819409 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:23.819530 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:23.819530 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:23.819702 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:23.819804 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:23.819806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:23.820444 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:23.820444 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:23.820981 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:23.820983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:23.821293 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:23.821294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:23.821342 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:23.821932 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:23.821937 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:23.823865 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":98,"threads_started":1}
W20260926 20:44:23.823995 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:23.819209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:49041 (local address 127.0.98.126:35517)
0926 20:44:23.819281 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:23.819283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:23.819405 (+ 122us) server_negotiation.cc:407] Connection header received
0926 20:44:23.819451 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:23.819452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:23.819464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:23.819511 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:23.819869 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:23.820379 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:23.821045 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:23.821231 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:23.821280 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:23.823948 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:49041: BlockingRecv error: recv got EOF from 127.0.98.67:49041 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 20:44:24 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:24 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:24.587116 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:24.582007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.65:49853)
0926 20:44:24.582164 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:24.582175 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:24.582222 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:24.582347 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:24.582348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:24.582514 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:24.582563 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:24.582565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:24.583348 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:24.583348 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:24.584027 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:24.584030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:24.584422 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:24.584423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:24.584464 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:24.584941 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:24.584945 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:24.586989 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
W20260926 20:44:24.587128 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:24.582093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:49853 (local address 127.0.98.126:35517)
0926 20:44:24.582216 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:24.582219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:24.582233 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:44:24.582259 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:24.582259 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:24.582270 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:24.582333 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:24.582638 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:24.583252 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:24.584095 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:24.584343 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:24.584423 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:24.587069 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:49853: BlockingRecv error: recv got EOF from 127.0.98.65:49853 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
Sep 26 20:44:24 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:24 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:24.713801 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:24.709140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.66:54655)
0926 20:44:24.709289 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:24.709302 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:24.709373 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:24.709500 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:24.709501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:24.709664 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:24.709710 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:24.709712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:24.710368 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:24.710369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:24.710912 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:24.710914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:24.711270 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:24.711271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:24.711310 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:24.711898 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:24.711904 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:24.713744 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
W20260926 20:44:24.713929 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:24.709203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:54655 (local address 127.0.98.126:35517)
0926 20:44:24.709292 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:24.709294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:24.709370 (+ 76us) server_negotiation.cc:407] Connection header received
0926 20:44:24.709432 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:24.709433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:24.709444 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:24.709486 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:24.709778 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:24.710281 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:24.710997 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:24.711200 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:24.711261 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:24.713849 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:54655: BlockingRecv error: recv got EOF from 127.0.98.66:54655 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 20:44:24 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:24 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:24.829010 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:24.824395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.67:34589)
0926 20:44:24.824536 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:24.824561 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:24.824640 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:24.824805 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:24.824806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:24.824982 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:24.825027 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:24.825029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:24.825716 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:24.825718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:24.826253 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:24.826254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:24.826596 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:24.826597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:24.826660 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:24.827144 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:24.827149 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:24.828915 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
W20260926 20:44:24.829063 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:24.824533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:34589 (local address 127.0.98.126:35517)
0926 20:44:24.824593 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:24.824595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:24.824621 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:44:24.824701 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:24.824702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:24.824716 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:24.824794 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:24.825141 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:24.825635 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:24.826333 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:24.826520 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:24.826594 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:24.828997 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:34589: BlockingRecv error: recv got EOF from 127.0.98.67:34589 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 20:44:25 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:25 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:25.592555 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:25.587644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.65:51615)
0926 20:44:25.587838 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:25.587850 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:25.587910 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:25.588111 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:25.588112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:25.588278 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:25.588330 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:25.588337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:25.589002 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:25.589003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:25.589549 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:25.589550 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:25.589930 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:25.589931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:25.590016 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:25.590602 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:25.590606 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:25.592496 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":133,"threads_started":1}
W20260926 20:44:25.592700 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:25.587767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:51615 (local address 127.0.98.126:35517)
0926 20:44:25.587889 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:25.587891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:25.587901 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:44:25.587968 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:25.587968 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:25.587979 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:25.588083 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:25.588428 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:25.588926 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:25.589654 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:25.589846 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:25.589911 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:25.592614 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:51615: BlockingRecv error: recv got EOF from 127.0.98.65:51615 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":54,"threads_started":1}
Sep 26 20:44:25 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:25 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:25.719692 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:25.714246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.66:48203)
0926 20:44:25.714368 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:25.714380 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:25.714461 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:25.714624 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:25.714625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:25.714797 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:25.714840 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:25.714842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:25.715622 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:25.715623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:25.716263 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:25.716265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:25.716782 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:25.716783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:25.716827 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:25.717365 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:25.717369 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:25.719498 (+ 2129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
W20260926 20:44:25.719763 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:25.714302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:48203 (local address 127.0.98.126:35517)
0926 20:44:25.714362 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:25.714364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:25.714442 (+ 78us) server_negotiation.cc:407] Connection header received
0926 20:44:25.714528 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:25.714529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:25.714540 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:25.714607 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:25.714924 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:25.715544 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:25.716358 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:25.716701 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:25.716787 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:25.719655 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:48203: BlockingRecv error: recv got EOF from 127.0.98.66:48203 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 20:44:25 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:25 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:25.834051 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:25.829478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.67:60377)
0926 20:44:25.829649 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:25.829660 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:25.829739 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:25.829869 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:25.829870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:25.830050 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:25.830094 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:25.830096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:25.830752 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:25.830753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:25.831275 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:25.831276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:25.831635 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:25.831636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:25.831695 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:25.832202 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:25.832207 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:25.833984 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":85,"threads_started":1}
W20260926 20:44:25.834231 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:25.829556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:60377 (local address 127.0.98.126:35517)
0926 20:44:25.829626 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:25.829629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:25.829723 (+ 94us) server_negotiation.cc:407] Connection header received
0926 20:44:25.829805 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:25.829806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:25.829818 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:25.829866 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:25.830181 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:25.830673 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:25.831355 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:25.831551 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:25.831620 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:25.834073 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:60377: BlockingRecv error: recv got EOF from 127.0.98.67:60377 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 20:44:26 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:26 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:26.598271 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:26.593116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.65:36293)
0926 20:44:26.593294 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:26.593305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:26.593370 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:26.593521 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:26.593522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:26.593697 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:26.593752 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:26.593754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:26.594421 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:26.594422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:26.594967 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:26.594969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:26.595397 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:26.595398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:26.595451 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:26.596109 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:26.596115 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:26.598179 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
W20260926 20:44:26.598366 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:26.593225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:36293 (local address 127.0.98.126:35517)
0926 20:44:26.593330 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:26.593331 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:26.593354 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:44:26.593429 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:26.593429 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:26.593440 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:26.593499 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:26.593816 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:26.594329 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:26.595096 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:26.595302 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:26.595368 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:26.598295 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:36293: BlockingRecv error: recv got EOF from 127.0.98.65:36293 (error 108)
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":50,"threads_started":1}
Sep 26 20:44:26 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:26 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:26.724902 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:26.720414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.66:45157)
0926 20:44:26.720543 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:26.720555 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:26.720624 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:26.720748 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:26.720749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:26.720928 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:26.720977 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:26.720979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:26.721600 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:26.721601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:26.722151 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:26.722153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:26.722512 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:26.722513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:26.722551 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:26.723013 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:26.723018 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:26.724829 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
W20260926 20:44:26.725052 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:26.720491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:45157 (local address 127.0.98.126:35517)
0926 20:44:26.720564 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:26.720566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:26.720631 (+ 65us) server_negotiation.cc:407] Connection header received
0926 20:44:26.720673 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:26.720673 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:26.720684 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:26.720735 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:26.721034 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:26.721530 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:26.722243 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:26.722440 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:26.722507 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:26.724970 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:45157: BlockingRecv error: recv got EOF from 127.0.98.66:45157 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 20:44:26 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:26 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:26.839329 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:26.834605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.67:36703)
0926 20:44:26.834753 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:26.834765 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:26.834831 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:26.834979 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:26.834980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:26.835220 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:26.835274 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:26.835276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:26.835980 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:26.835981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:26.836533 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:26.836535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:26.836917 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:26.836917 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:26.836976 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:26.837473 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:26.837478 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:26.839258 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
W20260926 20:44:26.839452 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:26.834651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:36703 (local address 127.0.98.126:35517)
0926 20:44:26.834731 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:26.834733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:26.834812 (+ 79us) server_negotiation.cc:407] Connection header received
0926 20:44:26.834892 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:26.834893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:26.834908 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:26.834974 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:26.835355 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:26.835905 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:26.836635 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:26.836828 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:26.836915 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:26.839348 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:36703: BlockingRecv error: recv got EOF from 127.0.98.67:36703 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 20:44:27 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:27 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:27.603600 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:27.598879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.65:36889)
0926 20:44:27.599025 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:27.599037 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:27.599096 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:27.599255 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:27.599256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:27.599425 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:27.599475 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:27.599476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:27.600186 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:27.600187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:27.600721 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:27.600723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:27.601079 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:27.601080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:27.601142 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:27.601623 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:27.601628 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:27.603524 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
W20260926 20:44:27.603713 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:27.598965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:36889 (local address 127.0.98.126:35517)
0926 20:44:27.599090 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:27.599092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:27.599102 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:44:27.599157 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:27.599158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:27.599168 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:27.599237 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:27.599538 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:27.600106 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:27.600802 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:27.600992 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:27.601047 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:27.603614 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:36889: BlockingRecv error: recv got EOF from 127.0.98.65:36889 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
Sep 26 20:44:27 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:27 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:27.730044 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:27.725365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.66:39781)
0926 20:44:27.725519 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:27.725533 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:27.725606 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:27.725763 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:27.725764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:27.725947 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:27.725992 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:27.725993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:27.726678 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:27.726679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:27.727227 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:27.727229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:27.727609 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:27.727610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:27.727664 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:27.728211 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:27.728217 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:27.729949 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
W20260926 20:44:27.730085 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:27.725487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:39781 (local address 127.0.98.126:35517)
0926 20:44:27.725543 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:27.725545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:27.725585 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:44:27.725690 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:27.725691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:27.725703 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:27.725749 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:27.726065 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:27.726566 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:27.727323 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:27.727529 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:27.727608 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:27.730025 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:39781: BlockingRecv error: recv got EOF from 127.0.98.66:39781 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 26 20:44:27 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:27 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:27.844696 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:27.839864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.67:35091)
0926 20:44:27.839990 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:27.840001 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:27.840078 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:27.840255 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:27.840256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:27.840424 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:27.840472 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:27.840473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:27.841102 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:27.841102 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:27.841626 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:27.841628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:27.841954 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:27.841955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:27.842009 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:27.842633 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:27.842638 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:27.844629 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
W20260926 20:44:27.844805 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:27.839892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:35091 (local address 127.0.98.126:35517)
0926 20:44:27.839962 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:27.839965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:27.840052 (+ 87us) server_negotiation.cc:407] Connection header received
0926 20:44:27.840141 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:27.840142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:27.840157 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:27.840217 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:27.840532 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:27.841030 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:27.841691 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:27.841869 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:27.841923 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:27.844730 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:35091: BlockingRecv error: recv got EOF from 127.0.98.67:35091 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 20:44:28 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:28 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:28.609172 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:28.604083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.65:54253)
0926 20:44:28.604259 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:28.604274 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:28.604343 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:28.604524 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:28.604525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:28.604690 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:28.604738 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:28.604740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:28.605423 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:28.605424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:28.605963 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:28.605965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:28.606369 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:28.606370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:28.606419 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:28.606955 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:28.606959 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:28.609061 (+ 2102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":73,"threads_started":1}
W20260926 20:44:28.609218 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:28.604185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:54253 (local address 127.0.98.126:35517)
0926 20:44:28.604338 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:28.604339 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:28.604350 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:44:28.604398 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:28.604398 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:28.604409 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:28.604477 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:28.604837 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:28.605344 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:28.606041 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:28.606278 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:28.606337 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:28.609147 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:54253: BlockingRecv error: recv got EOF from 127.0.98.65:54253 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
Sep 26 20:44:28 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:28 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:28.740362 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:28.730591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.66:37681)
0926 20:44:28.730739 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:28.730751 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:28.730857 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:28.730983 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:28.730984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:28.731186 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:28.731252 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:28.731255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:28.731937 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:28.731938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:28.732501 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:28.732503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:28.732858 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:28.732860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:28.732928 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:28.738354 (+ 5426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:28.738359 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:28.740295 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":207,"threads_started":1}
W20260926 20:44:28.740466 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:28.730668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:37681 (local address 127.0.98.126:35517)
0926 20:44:28.730728 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:28.730730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:28.730843 (+ 113us) server_negotiation.cc:407] Connection header received
0926 20:44:28.730908 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:28.730908 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:28.730918 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:28.730968 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:28.731315 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:28.731861 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:28.732568 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:28.732754 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:28.732830 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:28.740399 (+ 7569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:37681: BlockingRecv error: recv got EOF from 127.0.98.66:37681 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 26 20:44:28 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:28 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:28.850085 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:28.845176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.67:40403)
0926 20:44:28.845295 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:28.845307 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:28.845372 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:28.845511 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:28.845512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:28.845773 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:28.845833 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:28.845836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:28.846942 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:28.846943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:28.847467 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:28.847469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:28.847799 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:28.847800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:28.847841 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:28.848275 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:28.848279 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:28.850017 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
W20260926 20:44:28.850147 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:28.845242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:40403 (local address 127.0.98.126:35517)
0926 20:44:28.845290 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:28.845292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:28.845375 (+ 83us) server_negotiation.cc:407] Connection header received
0926 20:44:28.845417 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:28.845418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:28.845430 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:28.845482 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:28.845911 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:28.846853 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:28.847534 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:28.847707 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:28.847798 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:28.850096 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:40403: BlockingRecv error: recv got EOF from 127.0.98.67:40403 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 20:44:29 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:29 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:29.614517 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:29.609747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.65:55837)
0926 20:44:29.609922 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:29.609934 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:29.609993 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:29.610202 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:29.610203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:29.610365 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:29.610415 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:29.610417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:29.611053 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:29.611053 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:29.611584 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:29.611586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:29.612054 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:29.612055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:29.612103 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:29.612765 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:29.612770 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:29.614448 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":101,"threads_started":1}
W20260926 20:44:29.614662 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:29.609879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:55837 (local address 127.0.98.126:35517)
0926 20:44:29.609980 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:29.609982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:29.609992 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:44:29.610046 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:29.610046 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:29.610056 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:29.610162 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:29.610491 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:29.610987 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:29.611680 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:29.611965 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:29.612031 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:29.614565 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:55837: BlockingRecv error: recv got EOF from 127.0.98.65:55837 (error 108)
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":49,"threads_started":1}
Sep 26 20:44:29 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:44:29 dist-test-slave-4hd3 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:44:29.746312 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:29.740884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35517 (local address 127.0.98.66:52753)
0926 20:44:29.741023 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:29.741040 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:29.741091 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:29.741258 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:29.741260 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:29.741453 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:29.741502 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:29.741504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:29.742548 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:29.742549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:29.743312 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:29.743315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:29.743842 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:29.743843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:29.743899 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:29.744411 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:29.744415 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:29.746217 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:35517: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
W20260926 20:44:29.746438 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:29.740978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:52753 (local address 127.0.98.126:35517)
0926 20:44:29.741043 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:29.741046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:29.741079 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:44:29.741159 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:29.741160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:29.741179 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:29.741242 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:29.741578 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:29.742445 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:29.743387 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:29.743768 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:29.743846 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:29.746367 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:52753: BlockingRecv error: recv got EOF from 127.0.98.66:52753 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:44:29.768839 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 9113
I20260926 20:44:29.773799 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 9264
I20260926 20:44:29.778250 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 9396
I20260926 20:44:29.782287 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 9042
2026-09-26T20:44:29Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17400 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T20:44:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:44:29Z Disabled control of system clock
I20260926 20:44:29.809240 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36379
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:34155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:36379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 20:44:29.887936 9592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:29.888130 9592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:29.888152 9592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:29.889309 9592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:44:29.889351 9592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:29.889364 9592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:29.889379 9592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:44:29.889390 9592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:44:29.890635 9592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:36379
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:29.890825 9592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:29.891052 9592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:29.895658 9597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:29.895630 9600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:29.895680 9598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:29.895787 9592 server_base.cc:1034] running on GCE node
I20260926 20:44:29.896032 9592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:29.896365 9592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:29.897521 9592 hybrid_clock.cc:648] HybridClock initialized: now 1790455469897511 us; error 47 us; skew 500 ppm
I20260926 20:44:29.898684 9592 webserver.cc:460] Webserver started at http://127.0.98.126:44435/ using document root <none> and password file <none>
I20260926 20:44:29.898919 9592 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:29.898986 9592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:29.899148 9592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:29.900260 9592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "a58847146eff46b3b03a02542bcaf6ff"
format_stamp: "Formatted at 2026-09-26 20:44:29 on dist-test-slave-4hd3"
server_key: "d429d03351a990e78a26f45709125078"
server_key_iv: "1d52f07ef0455001ac0e151787f74cee"
server_key_version: "encryptionkey@0"
I20260926 20:44:29.900633 9592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "a58847146eff46b3b03a02542bcaf6ff"
format_stamp: "Formatted at 2026-09-26 20:44:29 on dist-test-slave-4hd3"
server_key: "d429d03351a990e78a26f45709125078"
server_key_iv: "1d52f07ef0455001ac0e151787f74cee"
server_key_version: "encryptionkey@0"
I20260926 20:44:29.901983 9592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:44:29.902724 9606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:29.902963 9592 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:29.903070 9592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "a58847146eff46b3b03a02542bcaf6ff"
format_stamp: "Formatted at 2026-09-26 20:44:29 on dist-test-slave-4hd3"
server_key: "d429d03351a990e78a26f45709125078"
server_key_iv: "1d52f07ef0455001ac0e151787f74cee"
server_key_version: "encryptionkey@0"
I20260926 20:44:29.903168 9592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:29.912853 9592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:29.913138 9592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:29.913278 9592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:29.917404 9592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:36379
I20260926 20:44:29.917441 9658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:36379 every 8 connection(s)
I20260926 20:44:29.917764 9592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:44:29.918416 9659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:29.921211 9659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff: Bootstrap starting.
I20260926 20:44:29.921826 9659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:29.922143 9659 log.cc:824] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff: Log is configured to *not* fsync() on all Append() calls
I20260926 20:44:29.922866 9659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff: No bootstrap required, opened a new log
I20260926 20:44:29.923123 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 9592
I20260926 20:44:29.923238 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:44:29.923439 393 external_mini_cluster.cc:1428] Setting key fe03fa197b83bacda00cde7d23387a52
I20260926 20:44:29.925099 9659 raft_consensus.cc:348] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58847146eff46b3b03a02542bcaf6ff" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36379 } }
I20260926 20:44:29.925272 9659 raft_consensus.cc:374] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:29.925344 9659 raft_consensus.cc:729] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58847146eff46b3b03a02542bcaf6ff, State: Initialized, Role: FOLLOWER
I20260926 20:44:29.925537 9659 consensus_queue.cc:260] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58847146eff46b3b03a02542bcaf6ff" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36379 } }
I20260926 20:44:29.925670 9659 raft_consensus.cc:388] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:44:29.925722 9659 raft_consensus.cc:482] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:44:29.925776 9659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:29.926345 9659 raft_consensus.cc:504] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58847146eff46b3b03a02542bcaf6ff" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36379 } }
I20260926 20:44:29.926468 9659 leader_election.cc:302] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a58847146eff46b3b03a02542bcaf6ff; no voters:
I20260926 20:44:29.926687 9659 leader_election.cc:288] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:44:29.926716 9664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:44:29.926892 9664 raft_consensus.cc:686] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff [term 1 LEADER]: Becoming Leader. State: Replica: a58847146eff46b3b03a02542bcaf6ff, State: Running, Role: LEADER
I20260926 20:44:29.927013 9659 sys_catalog.cc:564] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:44:29.927042 9664 consensus_queue.cc:237] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58847146eff46b3b03a02542bcaf6ff" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36379 } }
I20260926 20:44:29.928263 9666 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader a58847146eff46b3b03a02542bcaf6ff. Latest consensus state: current_term: 1 leader_uuid: "a58847146eff46b3b03a02542bcaf6ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58847146eff46b3b03a02542bcaf6ff" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36379 } } }
I20260926 20:44:29.928346 9665 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a58847146eff46b3b03a02542bcaf6ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58847146eff46b3b03a02542bcaf6ff" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36379 } } }
I20260926 20:44:29.928493 9665 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:29.928442 9666 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:29.928841 9677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:44:29.929394 9677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:44:29.930696 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:29.924039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49384 (local address 127.0.98.126:36379)
0926 20:44:29.924262 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:29.924275 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:29.924301 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:44:29.925075 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:29.925083 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:29.925153 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:29.925275 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:29.925612 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:29.927098 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:29.928263 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:29.928879 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:29.928996 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:29.929922 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:29.929925 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:29.929926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:44:29.930101 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:29.930137 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:44:29.930148 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:29.930209 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:29.930317 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 20:44:29.930432 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":108,"threads_started":1}
I20260926 20:44:29.931339 9677 catalog_manager.cc:1347] Generated new cluster ID: e09d671e37714b6ba1d9c1831082715f
I20260926 20:44:29.931396 9677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:44:29.952654 9677 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:44:29.953447 9677 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:44:29.969527 9677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a58847146eff46b3b03a02542bcaf6ff: Generated new TSK 0
I20260926 20:44:29.969787 9677 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:44:29.974373 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36379
--builtin_ntp_servers=127.0.98.84:34155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 20:44:30.055626 9683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:30.055951 9683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:30.055990 9683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:30.057192 9683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:30.057260 9683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:30.057300 9683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:44:30.058637 9683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:30.058926 9683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:30.059253 9683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:30.065229 9691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:30.065212 9689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:30.065357 9683 server_base.cc:1034] running on GCE node
W20260926 20:44:30.065202 9688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:30.065723 9683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:30.065972 9683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:30.067133 9683 hybrid_clock.cc:648] HybridClock initialized: now 1790455470067121 us; error 32 us; skew 500 ppm
I20260926 20:44:30.068427 9683 webserver.cc:460] Webserver started at http://127.0.98.65:36789/ using document root <none> and password file <none>
I20260926 20:44:30.068691 9683 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:30.068763 9683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:30.068939 9683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:30.070044 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "8afa66faebfb42e6931ee45ab6b74fe4"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "0b73e10094bf87f1eb5b2e2bc0ae3e95"
server_key_iv: "9e0448e061badbae401837ef65d6c3b8"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.070437 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "8afa66faebfb42e6931ee45ab6b74fe4"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "0b73e10094bf87f1eb5b2e2bc0ae3e95"
server_key_iv: "9e0448e061badbae401837ef65d6c3b8"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.071777 9683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:44:30.072546 9697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:30.072707 9683 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:44:30.072806 9683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "8afa66faebfb42e6931ee45ab6b74fe4"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "0b73e10094bf87f1eb5b2e2bc0ae3e95"
server_key_iv: "9e0448e061badbae401837ef65d6c3b8"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.072887 9683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:30.109530 9683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:30.109862 9683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:30.110018 9683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:30.110265 9683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:30.110599 9683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:30.110653 9683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:30.110687 9683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:30.110714 9683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:30.116443 9683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:37109
I20260926 20:44:30.116489 9810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:37109 every 8 connection(s)
I20260926 20:44:30.116801 9683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:44:30.119884 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 9683
I20260926 20:44:30.119956 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:44:30.120175 393 external_mini_cluster.cc:1428] Setting key 2159cb2abe95addbc1710401ea8414bf
I20260926 20:44:30.121196 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36379
--builtin_ntp_servers=127.0.98.84:34155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 20:44:30.122272 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:30.117480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:33687 (local address 127.0.98.126:36379)
0926 20:44:30.117535 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:30.117537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:30.117826 (+ 289us) server_negotiation.cc:407] Connection header received
0926 20:44:30.118300 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:30.118301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:30.118313 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:30.118372 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:30.119010 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:30.119498 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:30.120687 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:30.120991 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:30.121062 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:30.121973 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:30.121974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:30.121975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:44:30.121980 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:30.121997 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:44:30.122001 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:30.122058 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:30.122118 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 20:44:30.122153 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 20:44:30.122412 9813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:30.117497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36379 (local address 127.0.98.65:33687)
0926 20:44:30.117694 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:30.117727 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:30.118197 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:30.118425 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:30.118428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:30.118436 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:30.118910 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:30.118913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:30.119583 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:30.119583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:30.120619 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:30.120621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:30.121084 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:30.121085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:30.121194 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:30.121841 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:44:30.121842 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:30.121868 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:44:30.121889 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:30.122063 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:30.122066 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:30.122069 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:44:30.122120 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 20:44:30.122192 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":112,"threads_started":1}
I20260926 20:44:30.122952 9811 heartbeater.cc:347] Connected to a master server at 127.0.98.126:36379
I20260926 20:44:30.123049 9811 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:30.123221 9811 heartbeater.cc:511] Master 127.0.98.126:36379 requested a full tablet report, sending...
I20260926 20:44:30.123741 9623 ts_manager.cc:194] Registered new tserver with Master: 8afa66faebfb42e6931ee45ab6b74fe4 (127.0.98.65:37109)
I20260926 20:44:30.124336 9623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.65:33687
W20260926 20:44:30.202996 9814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:30.203274 9814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:30.203315 9814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:30.204497 9814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:30.204560 9814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:30.204600 9814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:44:30.206043 9814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:30.206297 9814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:30.206580 9814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:30.211476 9820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:30.211508 9819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:30.211619 9814 server_base.cc:1034] running on GCE node
W20260926 20:44:30.211728 9822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:30.212016 9814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:30.212280 9814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:30.213446 9814 hybrid_clock.cc:648] HybridClock initialized: now 1790455470213432 us; error 42 us; skew 500 ppm
I20260926 20:44:30.214789 9814 webserver.cc:460] Webserver started at http://127.0.98.66:46811/ using document root <none> and password file <none>
I20260926 20:44:30.215029 9814 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:30.215099 9814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:30.215256 9814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:30.216361 9814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "146d21563ee143eca279c6abd8cd3b7a"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "f8f7288010f5f4c3cd44ace874dd0882"
server_key_iv: "d8c8ef1143a9ff3dc3ac413c949e026f"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.216714 9814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "146d21563ee143eca279c6abd8cd3b7a"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "f8f7288010f5f4c3cd44ace874dd0882"
server_key_iv: "d8c8ef1143a9ff3dc3ac413c949e026f"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.217952 9814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:44:30.218643 9828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:30.218837 9814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:44:30.218935 9814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "146d21563ee143eca279c6abd8cd3b7a"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "f8f7288010f5f4c3cd44ace874dd0882"
server_key_iv: "d8c8ef1143a9ff3dc3ac413c949e026f"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.219008 9814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:30.226075 9814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:30.226341 9814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:30.226482 9814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:30.226687 9814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:30.226992 9814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:30.227043 9814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:30.227082 9814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:30.227109 9814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:30.232895 9814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:42047
I20260926 20:44:30.232963 9941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:42047 every 8 connection(s)
I20260926 20:44:30.233417 9814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:44:30.236168 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 9814
I20260926 20:44:30.236269 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:44:30.236464 393 external_mini_cluster.cc:1428] Setting key d2dd02aa3adfdee9e76e86c25ef722a8
I20260926 20:44:30.237435 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36379
--builtin_ntp_servers=127.0.98.84:34155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 20:44:30.238950 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:30.234174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:45897 (local address 127.0.98.126:36379)
0926 20:44:30.234242 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:30.234244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:30.234489 (+ 245us) server_negotiation.cc:407] Connection header received
0926 20:44:30.235024 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:30.235025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:30.235038 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:30.235102 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:30.235840 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:30.236431 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:30.237524 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:30.237745 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:30.237817 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:30.238714 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:30.238715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:30.238716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:44:30.238720 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:30.238735 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:44:30.238739 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:30.238790 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:30.238868 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 20:44:30.238891 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:44:30.239223 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:30.234156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36379 (local address 127.0.98.66:45897)
0926 20:44:30.234366 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:30.234400 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:30.234915 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:30.235177 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:30.235181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:30.235189 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:30.235683 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:30.235687 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:30.236522 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:30.236523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:30.237425 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:30.237427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:30.237835 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:30.237836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:30.237961 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:30.238546 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:44:30.238547 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:30.238589 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:44:30.238623 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:30.238792 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:30.238796 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:30.238799 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:44:30.238872 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 20:44:30.238984 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":98,"threads_started":1}
I20260926 20:44:30.239806 9942 heartbeater.cc:347] Connected to a master server at 127.0.98.126:36379
I20260926 20:44:30.239894 9942 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:30.240073 9942 heartbeater.cc:511] Master 127.0.98.126:36379 requested a full tablet report, sending...
I20260926 20:44:30.240447 9623 ts_manager.cc:194] Registered new tserver with Master: 146d21563ee143eca279c6abd8cd3b7a (127.0.98.66:42047)
I20260926 20:44:30.240823 9623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.66:45897
W20260926 20:44:30.318832 9945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:30.319080 9945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:30.319111 9945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:30.320298 9945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:30.320359 9945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:30.320402 9945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:44:30.321838 9945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:30.322091 9945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:30.322441 9945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:30.327375 9950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:30.327368 9951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:30.327368 9953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:30.327834 9945 server_base.cc:1034] running on GCE node
I20260926 20:44:30.328003 9945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:30.328231 9945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:30.329376 9945 hybrid_clock.cc:648] HybridClock initialized: now 1790455470329364 us; error 36 us; skew 500 ppm
I20260926 20:44:30.330670 9945 webserver.cc:460] Webserver started at http://127.0.98.67:33221/ using document root <none> and password file <none>
I20260926 20:44:30.330926 9945 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:30.330976 9945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:30.331171 9945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:30.332365 9945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "72d47b1700b04d56b1a0a25478e47c6c"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "8782962179b52317f301db8d3267b8b9"
server_key_iv: "d62a5a92083570cea273759a4f092f90"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.332731 9945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "72d47b1700b04d56b1a0a25478e47c6c"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "8782962179b52317f301db8d3267b8b9"
server_key_iv: "d62a5a92083570cea273759a4f092f90"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.334044 9945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:44:30.334892 9959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:30.335125 9945 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:30.335199 9945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "72d47b1700b04d56b1a0a25478e47c6c"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "8782962179b52317f301db8d3267b8b9"
server_key_iv: "d62a5a92083570cea273759a4f092f90"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.335290 9945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:30.344967 9945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:30.345352 9945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:30.345512 9945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:30.345767 9945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:30.346095 9945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:30.346161 9945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:30.346203 9945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:30.346233 9945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:30.352077 9945 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:36515
I20260926 20:44:30.352154 10072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:36515 every 8 connection(s)
I20260926 20:44:30.352522 9945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:44:30.358989 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:30.354407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:52035 (local address 127.0.98.126:36379)
0926 20:44:30.354460 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:30.354462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:30.354474 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:44:30.355008 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:30.355009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:30.355022 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:30.355076 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:30.356001 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:30.356795 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:30.357754 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:30.357956 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:30.358014 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:30.358790 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:30.358791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:30.358791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:44:30.358795 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:30.358809 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:44:30.358813 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:30.358857 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:30.358906 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 20:44:30.358928 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:44:30.359228 10075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:30.353946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36379 (local address 127.0.98.67:52035)
0926 20:44:30.354157 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:30.354199 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:30.354883 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:30.355146 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:30.355150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:30.355161 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:30.355864 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:30.355871 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:30.356888 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:30.356889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:30.357678 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:30.357680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:30.358034 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:30.358035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:30.358140 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:30.358652 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:44:30.358653 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:30.358680 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:44:30.358702 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:30.358857 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:30.358859 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:30.358861 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:44:30.358907 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 20:44:30.358983 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
I20260926 20:44:30.359771 10073 heartbeater.cc:347] Connected to a master server at 127.0.98.126:36379
I20260926 20:44:30.359886 10073 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:30.360064 10073 heartbeater.cc:511] Master 127.0.98.126:36379 requested a full tablet report, sending...
I20260926 20:44:30.360460 9623 ts_manager.cc:194] Registered new tserver with Master: 72d47b1700b04d56b1a0a25478e47c6c (127.0.98.67:36515)
I20260926 20:44:30.360785 9623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.67:52035
I20260926 20:44:30.362624 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 9945
I20260926 20:44:30.362701 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:44:30.362849 393 external_mini_cluster.cc:1428] Setting key ada8bc0b539f093dd92bf1a7184d9293
I20260926 20:44:30.363659 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 20:44:30.365571 9663 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:30.365035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.0.98.126:36379)
0926 20:44:30.365077 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:30.365079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:30.365195 (+ 116us) server_negotiation.cc:407] Connection header received
0926 20:44:30.365271 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:30.365272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:30.365284 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:30.365363 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:30.365502 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49394: BlockingRecv error: recv got EOF from 127.0.0.1:49394 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 20:44:30.365633 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 9683
I20260926 20:44:30.370537 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 9814
I20260926 20:44:30.374997 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 9945
I20260926 20:44:30.379055 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 9592
2026-09-26T20:44:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (596 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T20:44:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:44:30Z Disabled control of system clock
I20260926 20:44:30.406728 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36017
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:41365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:36017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 20:44:30.486032 10084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:30.486243 10084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:30.486263 10084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:30.487351 10084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:44:30.487386 10084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:30.487398 10084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:30.487411 10084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:44:30.487423 10084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:44:30.488708 10084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:36017
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:30.488906 10084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:30.489135 10084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:30.493872 10090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:30.493891 10092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:30.493892 10089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:30.493968 10084 server_base.cc:1034] running on GCE node
I20260926 20:44:30.494313 10084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:30.494603 10084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:30.495784 10084 hybrid_clock.cc:648] HybridClock initialized: now 1790455470495756 us; error 41 us; skew 500 ppm
I20260926 20:44:30.496990 10084 webserver.cc:460] Webserver started at http://127.0.98.126:45409/ using document root <none> and password file <none>
I20260926 20:44:30.497220 10084 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:30.497268 10084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:30.497421 10084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:30.498509 10084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "a56bac97a5394b59bbeb250a9358d113"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "91b4b8bdff3abf9bdcf8c7c69bf9d793"
server_key_iv: "a4beb5be22ce6ea60c54b1d3a9b4c60c"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.498862 10084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "a56bac97a5394b59bbeb250a9358d113"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "91b4b8bdff3abf9bdcf8c7c69bf9d793"
server_key_iv: "a4beb5be22ce6ea60c54b1d3a9b4c60c"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.500227 10084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:44:30.500993 10098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:30.501192 10084 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:44:30.501286 10084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "a56bac97a5394b59bbeb250a9358d113"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "91b4b8bdff3abf9bdcf8c7c69bf9d793"
server_key_iv: "a4beb5be22ce6ea60c54b1d3a9b4c60c"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.501362 10084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:30.504593 10084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:30.504843 10084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:30.504981 10084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:30.509037 10084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:36017
I20260926 20:44:30.509068 10150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:36017 every 8 connection(s)
I20260926 20:44:30.509393 10084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:44:30.510062 10151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:30.510584 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 10084
I20260926 20:44:30.510679 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:44:30.510876 393 external_mini_cluster.cc:1428] Setting key bb9e9297d51095b1f6d2edecb1d3fdb9
I20260926 20:44:30.513455 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:30.511402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45920 (local address 127.0.98.126:36017)
0926 20:44:30.511620 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:30.511629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:30.511661 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:44:30.512170 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:30.512178 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:30.512234 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:30.512378 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:30.513007 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:30.513010 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:30.513010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:44:30.513015 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:30.513065 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:44:30.513078 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:30.513161 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:30.513225 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 20:44:30.513279 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260926 20:44:30.514000 10151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113: Bootstrap starting.
I20260926 20:44:30.514619 10151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:30.514899 10151 log.cc:824] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113: Log is configured to *not* fsync() on all Append() calls
I20260926 20:44:30.515610 10151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113: No bootstrap required, opened a new log
I20260926 20:44:30.517004 10151 raft_consensus.cc:348] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56bac97a5394b59bbeb250a9358d113" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36017 } }
I20260926 20:44:30.517134 10151 raft_consensus.cc:374] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:30.517159 10151 raft_consensus.cc:729] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a56bac97a5394b59bbeb250a9358d113, State: Initialized, Role: FOLLOWER
I20260926 20:44:30.517304 10151 consensus_queue.cc:260] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56bac97a5394b59bbeb250a9358d113" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36017 } }
I20260926 20:44:30.517390 10151 raft_consensus.cc:388] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:44:30.517413 10151 raft_consensus.cc:482] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:44:30.517467 10151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:30.517974 10151 raft_consensus.cc:504] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56bac97a5394b59bbeb250a9358d113" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36017 } }
I20260926 20:44:30.518071 10151 leader_election.cc:302] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a56bac97a5394b59bbeb250a9358d113; no voters:
I20260926 20:44:30.518256 10151 leader_election.cc:288] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:44:30.518556 10151 sys_catalog.cc:564] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:44:30.518589 10156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:44:30.518740 10156 raft_consensus.cc:686] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113 [term 1 LEADER]: Becoming Leader. State: Replica: a56bac97a5394b59bbeb250a9358d113, State: Running, Role: LEADER
I20260926 20:44:30.518834 10156 consensus_queue.cc:237] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56bac97a5394b59bbeb250a9358d113" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36017 } }
I20260926 20:44:30.519289 10157 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a56bac97a5394b59bbeb250a9358d113" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56bac97a5394b59bbeb250a9358d113" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36017 } } }
I20260926 20:44:30.519299 10158 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a56bac97a5394b59bbeb250a9358d113. Latest consensus state: current_term: 1 leader_uuid: "a56bac97a5394b59bbeb250a9358d113" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56bac97a5394b59bbeb250a9358d113" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36017 } } }
I20260926 20:44:30.519407 10157 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113 [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:30.519411 10158 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113 [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:30.519593 10165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:44:30.520220 10165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:44:30.521620 10165 catalog_manager.cc:1347] Generated new cluster ID: fcc0a528cb86407c83d01e348c565c66
I20260926 20:44:30.521670 10165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:44:30.531876 10165 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:44:30.532060 10165 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:44:30.541260 10165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a56bac97a5394b59bbeb250a9358d113: Generated new TSK 0
I20260926 20:44:30.541461 10165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:44:30.544095 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36017
--builtin_ntp_servers=127.0.98.84:41365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 20:44:30.624230 10175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:30.624426 10175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:30.624447 10175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:30.625609 10175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:30.625649 10175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:30.625674 10175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:44:30.627023 10175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:30.627262 10175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:30.627574 10175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:30.633324 10181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:30.633368 10180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:30.633363 10183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:30.633590 10175 server_base.cc:1034] running on GCE node
I20260926 20:44:30.634023 10175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:30.634305 10175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:30.635485 10175 hybrid_clock.cc:648] HybridClock initialized: now 1790455470635468 us; error 34 us; skew 500 ppm
I20260926 20:44:30.636997 10175 webserver.cc:460] Webserver started at http://127.0.98.65:35575/ using document root <none> and password file <none>
I20260926 20:44:30.637329 10175 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:30.637405 10175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:30.637535 10175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:30.638912 10175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "deaa0d2ef422490aaa7ec072912b7481"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "46218541a8fa63c33bb484048fae28ae"
server_key_iv: "ffa23140ecf11bc38e263a83b73eb6ed"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.639307 10175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "deaa0d2ef422490aaa7ec072912b7481"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "46218541a8fa63c33bb484048fae28ae"
server_key_iv: "ffa23140ecf11bc38e263a83b73eb6ed"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.640856 10175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260926 20:44:30.641696 10189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:30.641913 10175 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:30.642001 10175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "deaa0d2ef422490aaa7ec072912b7481"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "46218541a8fa63c33bb484048fae28ae"
server_key_iv: "ffa23140ecf11bc38e263a83b73eb6ed"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.642071 10175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:30.646679 10175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:30.646946 10175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:30.647064 10175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:30.647255 10175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:30.647547 10175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:30.647584 10175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:30.647609 10175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:30.647645 10175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:30.653630 10175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:46541
I20260926 20:44:30.653661 10302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:46541 every 8 connection(s)
I20260926 20:44:30.653980 10175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:44:30.656888 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:30.654726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:50393 (local address 127.0.98.126:36017)
0926 20:44:30.654784 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:30.654786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:30.654996 (+ 210us) server_negotiation.cc:407] Connection header received
0926 20:44:30.655467 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:30.655468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:30.655474 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:30.655539 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:30.656691 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:30.656693 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:30.656693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:44:30.656697 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:30.656711 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:44:30.656715 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:30.656762 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:30.656825 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 20:44:30.656837 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 20:44:30.657066 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:30.654678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36017 (local address 127.0.98.65:50393)
0926 20:44:30.654863 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:30.654903 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:30.655349 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:30.655609 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:30.655613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:30.655620 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:30.656546 (+ 926us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:44:30.656546 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:30.656575 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:44:30.656603 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:30.656761 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:30.656763 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:30.656766 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:44:30.656818 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 20:44:30.656865 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
I20260926 20:44:30.657594 10303 heartbeater.cc:347] Connected to a master server at 127.0.98.126:36017
I20260926 20:44:30.657670 10303 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:30.657830 10303 heartbeater.cc:511] Master 127.0.98.126:36017 requested a full tablet report, sending...
I20260926 20:44:30.658345 10115 ts_manager.cc:194] Registered new tserver with Master: deaa0d2ef422490aaa7ec072912b7481 (127.0.98.65:46541)
I20260926 20:44:30.658646 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 10175
I20260926 20:44:30.658743 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:44:30.658927 393 external_mini_cluster.cc:1428] Setting key 6c0baf6b82d049e9119eae2ea5840284
I20260926 20:44:30.659863 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36017
--builtin_ntp_servers=127.0.98.84:41365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 20:44:30.739341 10306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:30.739560 10306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:30.739583 10306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:30.740741 10306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:30.740779 10306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:30.740804 10306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:44:30.742071 10306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:30.742277 10306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:30.742517 10306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:30.747548 10311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:30.747536 10312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:30.747534 10314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:30.747601 10306 server_base.cc:1034] running on GCE node
I20260926 20:44:30.747952 10306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:30.748273 10306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:30.749429 10306 hybrid_clock.cc:648] HybridClock initialized: now 1790455470749401 us; error 40 us; skew 500 ppm
I20260926 20:44:30.750748 10306 webserver.cc:460] Webserver started at http://127.0.98.66:40465/ using document root <none> and password file <none>
I20260926 20:44:30.751029 10306 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:30.751099 10306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:30.751247 10306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:30.752403 10306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "e6cfe147047b43dea0e35c77c7371889"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "496f5c33ae5378f24f289542f49b6d34"
server_key_iv: "64f06ab7f5c82a8dfdee1f870451f262"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.752763 10306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "e6cfe147047b43dea0e35c77c7371889"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "496f5c33ae5378f24f289542f49b6d34"
server_key_iv: "64f06ab7f5c82a8dfdee1f870451f262"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.754070 10306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:44:30.755004 10320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:30.755203 10306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:30.755301 10306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "e6cfe147047b43dea0e35c77c7371889"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "496f5c33ae5378f24f289542f49b6d34"
server_key_iv: "64f06ab7f5c82a8dfdee1f870451f262"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.755376 10306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:30.758829 10306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:30.759101 10306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:30.759229 10306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:30.759446 10306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:30.759789 10306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:30.759848 10306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:30.759887 10306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:30.759917 10306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:30.765825 10306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:35681
I20260926 20:44:30.765862 10433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:35681 every 8 connection(s)
I20260926 20:44:30.766187 10306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:44:30.769097 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:30.766927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:58881 (local address 127.0.98.126:36017)
0926 20:44:30.766969 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:30.766971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:30.767224 (+ 253us) server_negotiation.cc:407] Connection header received
0926 20:44:30.767653 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:30.767654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:30.767659 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:30.767754 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:30.768922 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:30.768923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:30.768923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:44:30.768926 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:30.768936 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:44:30.768939 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:30.768981 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:30.769032 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 20:44:30.769043 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:44:30.769339 10436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:30.766915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36017 (local address 127.0.98.66:58881)
0926 20:44:30.767073 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:30.767105 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:30.767573 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:30.767826 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:30.767829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:30.767838 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:30.768795 (+ 957us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:44:30.768796 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:30.768824 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:44:30.768851 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:30.768982 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:30.768984 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:30.768987 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:44:30.769040 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 20:44:30.769087 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":82,"threads_started":1}
I20260926 20:44:30.769770 10434 heartbeater.cc:347] Connected to a master server at 127.0.98.126:36017
I20260926 20:44:30.769847 10434 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:30.770012 10434 heartbeater.cc:511] Master 127.0.98.126:36017 requested a full tablet report, sending...
I20260926 20:44:30.770378 10115 ts_manager.cc:194] Registered new tserver with Master: e6cfe147047b43dea0e35c77c7371889 (127.0.98.66:35681)
I20260926 20:44:30.774471 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 10306
I20260926 20:44:30.774565 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:44:30.774811 393 external_mini_cluster.cc:1428] Setting key 63457619847952d86502bf68deb1471e
I20260926 20:44:30.775758 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36017
--builtin_ntp_servers=127.0.98.84:41365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 20:44:30.856169 10437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:30.856370 10437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:30.856393 10437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:30.857487 10437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:30.857520 10437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:30.857545 10437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:44:30.858901 10437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:30.859194 10437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:30.859493 10437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:30.864384 10443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:30.864392 10445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:30.864616 10437 server_base.cc:1034] running on GCE node
W20260926 20:44:30.864647 10442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:30.864866 10437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:30.865156 10437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:30.866390 10437 hybrid_clock.cc:648] HybridClock initialized: now 1790455470866382 us; error 34 us; skew 500 ppm
I20260926 20:44:30.867691 10437 webserver.cc:460] Webserver started at http://127.0.98.67:43737/ using document root <none> and password file <none>
I20260926 20:44:30.867987 10437 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:30.868059 10437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:30.868232 10437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:30.869287 10437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "873a7edba1c340238fd7335870d37dde"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "f4754876a83a38c4d7a483c413f9f3b9"
server_key_iv: "ff549a1f000fa42fac5516820f566e14"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.869655 10437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "873a7edba1c340238fd7335870d37dde"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "f4754876a83a38c4d7a483c413f9f3b9"
server_key_iv: "ff549a1f000fa42fac5516820f566e14"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.871068 10437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:44:30.871986 10451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:30.872202 10437 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:44:30.872303 10437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "873a7edba1c340238fd7335870d37dde"
format_stamp: "Formatted at 2026-09-26 20:44:30 on dist-test-slave-4hd3"
server_key: "f4754876a83a38c4d7a483c413f9f3b9"
server_key_iv: "ff549a1f000fa42fac5516820f566e14"
server_key_version: "encryptionkey@0"
I20260926 20:44:30.872391 10437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:30.875754 10437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:30.876012 10437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:30.876194 10437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:30.876400 10437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:30.876698 10437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:30.876750 10437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:30.876788 10437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:30.876816 10437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:30.882369 10437 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:42501
I20260926 20:44:30.882408 10564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:42501 every 8 connection(s)
I20260926 20:44:30.882699 10437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:44:30.885846 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:30.883435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:54563 (local address 127.0.98.126:36017)
0926 20:44:30.883513 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:30.883516 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:30.883753 (+ 237us) server_negotiation.cc:407] Connection header received
0926 20:44:30.884314 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:30.884316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:30.884322 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:30.884377 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:30.885551 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:30.885553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:30.885553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:44:30.885558 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:30.885573 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:44:30.885578 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:30.885634 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:30.885754 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 20:44:30.885771 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:44:30.886044 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:30.883406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36017 (local address 127.0.98.67:54563)
0926 20:44:30.883603 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:30.883652 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:30.884185 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:30.884449 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:30.884453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:30.884464 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:30.885385 (+ 921us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:44:30.885385 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:30.885414 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:44:30.885443 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:30.885659 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:30.885663 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:30.885666 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:44:30.885766 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 20:44:30.885829 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260926 20:44:30.886551 10565 heartbeater.cc:347] Connected to a master server at 127.0.98.126:36017
I20260926 20:44:30.886629 10565 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:30.886801 10565 heartbeater.cc:511] Master 127.0.98.126:36017 requested a full tablet report, sending...
I20260926 20:44:30.887213 10115 ts_manager.cc:194] Registered new tserver with Master: 873a7edba1c340238fd7335870d37dde (127.0.98.67:42501)
I20260926 20:44:30.890339 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 10437
I20260926 20:44:30.890413 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:44:30.890551 393 external_mini_cluster.cc:1428] Setting key de5f625c821012eefd8ea9ee39d3d993
I20260926 20:44:30.891445 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 20:44:30.893237 10154 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:44:30.892803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45934 (local address 127.0.98.126:36017)
0926 20:44:30.892862 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:30.892864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:30.892992 (+ 128us) server_negotiation.cc:407] Connection header received
0926 20:44:30.893041 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:30.893042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:30.893045 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:30.893093 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:30.893198 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45934: BlockingRecv error: recv got EOF from 127.0.0.1:45934 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:44:30.893404 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 10175
I20260926 20:44:30.898103 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 10306
I20260926 20:44:30.902797 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 10437
I20260926 20:44:30.908172 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 10084
2026-09-26T20:44:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (529 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:44:30 dist-test-slave-4hd3 krb5kdc[10575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:44:30 dist-test-slave-4hd3 krb5kdc[10575](info): set up 2 sockets
Sep 26 20:44:30 dist-test-slave-4hd3 krb5kdc[10575](info): commencing operation
krb5kdc: starting...
W20260926 20:44:31.948854 393 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:44:32 dist-test-slave-4hd3 krb5kdc[10575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:44:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:44:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:44:32.065878 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34231
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:34231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:32.146746 10588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:32.146950 10588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:32.146970 10588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:32.148145 10588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:44:32.148185 10588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:32.148200 10588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:32.148213 10588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:44:32.148226 10588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:44:32.149688 10588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:34231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:32.149925 10588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:32.150173 10588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:32.154829 10593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:32.154897 10594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:32.154841 10596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:32.155090 10588 server_base.cc:1034] running on GCE node
I20260926 20:44:32.155306 10588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:32.155529 10588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:32.156662 10588 hybrid_clock.cc:648] HybridClock initialized: now 1790455472156650 us; error 28 us; skew 500 ppm
Sep 26 20:44:32 dist-test-slave-4hd3 krb5kdc[10575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:32.159042 10588 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:44:32.159513 10588 webserver.cc:460] Webserver started at http://127.0.98.126:41331/ using document root <none> and password file <none>
I20260926 20:44:32.159780 10588 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:32.159847 10588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:32.159998 10588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:32.161020 10588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "55a9a6b50fcf42fba376d35e61841f62"
format_stamp: "Formatted at 2026-09-26 20:44:32 on dist-test-slave-4hd3"
server_key: "da817ce72fcd0ec0260f16a633eb5bbc"
server_key_iv: "65fec935155668da2dd3812dd6877b83"
server_key_version: "encryptionkey@0"
I20260926 20:44:32.161386 10588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "55a9a6b50fcf42fba376d35e61841f62"
format_stamp: "Formatted at 2026-09-26 20:44:32 on dist-test-slave-4hd3"
server_key: "da817ce72fcd0ec0260f16a633eb5bbc"
server_key_iv: "65fec935155668da2dd3812dd6877b83"
server_key_version: "encryptionkey@0"
I20260926 20:44:32.162662 10588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:44:32.163339 10603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:32.163511 10588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:44:32.163605 10588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "55a9a6b50fcf42fba376d35e61841f62"
format_stamp: "Formatted at 2026-09-26 20:44:32 on dist-test-slave-4hd3"
server_key: "da817ce72fcd0ec0260f16a633eb5bbc"
server_key_iv: "65fec935155668da2dd3812dd6877b83"
server_key_version: "encryptionkey@0"
I20260926 20:44:32.163692 10588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:32.180423 10588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:32.182720 10588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:32.182885 10588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:32.186860 10588 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:34231
I20260926 20:44:32.186895 10655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:34231 every 8 connection(s)
I20260926 20:44:32.187227 10588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:44:32.188271 10656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:32.190392 10656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62: Bootstrap starting.
I20260926 20:44:32.190609 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 10588
I20260926 20:44:32.190707 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:44:32.190899 393 external_mini_cluster.cc:1428] Setting key f0ab56cd05e724ea0c253c8c19c17196
I20260926 20:44:32.191094 10656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:32.191423 10656 log.cc:824] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62: Log is configured to *not* fsync() on all Append() calls
I20260926 20:44:32.192194 10656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62: No bootstrap required, opened a new log
I20260926 20:44:32.194432 10656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a9a6b50fcf42fba376d35e61841f62" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34231 } }
I20260926 20:44:32.194598 10656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:32.194638 10656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a9a6b50fcf42fba376d35e61841f62, State: Initialized, Role: FOLLOWER
I20260926 20:44:32.194768 10656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a9a6b50fcf42fba376d35e61841f62" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34231 } }
I20260926 20:44:32.194854 10656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:44:32.194876 10656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:44:32.194947 10656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:32.195497 10656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a9a6b50fcf42fba376d35e61841f62" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34231 } }
I20260926 20:44:32.195593 10656 leader_election.cc:302] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55a9a6b50fcf42fba376d35e61841f62; no voters:
I20260926 20:44:32.195758 10656 leader_election.cc:288] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:44:32.195808 10661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:44:32.195956 10661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62 [term 1 LEADER]: Becoming Leader. State: Replica: 55a9a6b50fcf42fba376d35e61841f62, State: Running, Role: LEADER
I20260926 20:44:32.195986 10656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:44:32.196100 10661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a9a6b50fcf42fba376d35e61841f62" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34231 } }
I20260926 20:44:32.196506 10662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55a9a6b50fcf42fba376d35e61841f62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a9a6b50fcf42fba376d35e61841f62" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34231 } } }
I20260926 20:44:32.196705 10662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62 [sys.catalog]: This master's current role is: LEADER
I20260926 20:44:32.196640 10663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55a9a6b50fcf42fba376d35e61841f62. Latest consensus state: current_term: 1 leader_uuid: "55a9a6b50fcf42fba376d35e61841f62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a9a6b50fcf42fba376d35e61841f62" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34231 } } }
I20260926 20:44:32.196856 10663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62 [sys.catalog]: This master's current role is: LEADER
Sep 26 20:44:32 dist-test-slave-4hd3 krb5kdc[10575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
W20260926 20:44:32.199162 10677 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:44:32.199221 10677 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:44:32.199267 10667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:44:32.199425 10667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:44:32.201274 10667 catalog_manager.cc:1347] Generated new cluster ID: 6943dd4f96344ef293e8cf6f935234d9
I20260926 20:44:32.201325 10667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:44:32.202443 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.191390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49594 (local address 127.0.98.126:34231)
0926 20:44:32.191624 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:32.191633 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:32.191669 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:44:32.192277 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:32.192287 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:32.192354 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:32.192491 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:32.193072 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.194501 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.195424 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.196055 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.196471 (+ 416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.198710 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:32.198713 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:32.198714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:32.198724 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:32.200824 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:32.201182 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:32.201184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:32.201185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:32.201310 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:32.201528 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:32.201529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:32.201529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:32.201823 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:32.201965 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:32.202090 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 20:44:32.202224 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":84,"threads_started":1}
I20260926 20:44:32.232072 10667 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:44:32.232620 10667 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:44:32.238464 10667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62: Generated new TSK 0
I20260926 20:44:32.238658 10667 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260926 20:44:32.270700 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34231
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:32.350674 10682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:32.350868 10682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:32.350890 10682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:32.352195 10682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:32.352304 10682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:32.352371 10682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:44:32.353840 10682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:32.354082 10682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:32.354315 10682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:32.359329 10687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:32.359391 10682 server_base.cc:1034] running on GCE node
W20260926 20:44:32.359350 10688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:32.359341 10690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:32.359791 10682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:32.360052 10682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:32.361202 10682 hybrid_clock.cc:648] HybridClock initialized: now 1790455472361180 us; error 37 us; skew 500 ppm
Sep 26 20:44:32 dist-test-slave-4hd3 krb5kdc[10575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:32.364472 10682 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:44:32.364993 10682 webserver.cc:460] Webserver started at http://127.0.98.65:43865/ using document root <none> and password file <none>
I20260926 20:44:32.365247 10682 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:32.365315 10682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:32.365449 10682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:32.366626 10682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "c9d670f0a64448df944937aae3cf1cfa"
format_stamp: "Formatted at 2026-09-26 20:44:32 on dist-test-slave-4hd3"
server_key: "898a892dd58ba7b9b3960bc13b15ff3f"
server_key_iv: "847eadadb0717d19b1f232d607b8bf85"
server_key_version: "encryptionkey@0"
I20260926 20:44:32.366936 10682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "c9d670f0a64448df944937aae3cf1cfa"
format_stamp: "Formatted at 2026-09-26 20:44:32 on dist-test-slave-4hd3"
server_key: "898a892dd58ba7b9b3960bc13b15ff3f"
server_key_iv: "847eadadb0717d19b1f232d607b8bf85"
server_key_version: "encryptionkey@0"
I20260926 20:44:32.368343 10682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:44:32.369110 10697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:32.369359 10682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:44:32.369449 10682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "c9d670f0a64448df944937aae3cf1cfa"
format_stamp: "Formatted at 2026-09-26 20:44:32 on dist-test-slave-4hd3"
server_key: "898a892dd58ba7b9b3960bc13b15ff3f"
server_key_iv: "847eadadb0717d19b1f232d607b8bf85"
server_key_version: "encryptionkey@0"
I20260926 20:44:32.369503 10682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:32.395128 10682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:32.397013 10682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:32.397171 10682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:32.397435 10682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:32.397794 10682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:32.397828 10682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:32.397850 10682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:32.397886 10682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:32.403549 10682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:33673
I20260926 20:44:32.403625 10810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:33673 every 8 connection(s)
I20260926 20:44:32.403960 10682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:44:32.405402 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 10682
I20260926 20:44:32.405501 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:44:32.405721 393 external_mini_cluster.cc:1428] Setting key a3a0a307ffa18d9399bc21eb113fd515
Sep 26 20:44:32 dist-test-slave-4hd3 krb5kdc[10575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260926 20:44:32.413888 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.404675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:43129 (local address 127.0.98.126:34231)
0926 20:44:32.404736 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:32.404738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:32.404943 (+ 205us) server_negotiation.cc:407] Connection header received
0926 20:44:32.405614 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:32.405615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:32.405626 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:32.405689 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:32.406801 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.407287 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.408472 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.408719 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.408774 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.411114 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:32.411116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:32.411116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:32.411124 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:32.412539 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:32.412946 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:32.412947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:32.412948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:32.413004 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:32.413257 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:32.413257 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:32.413258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:32.413364 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:32.413432 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:32.413721 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 20:44:32.413799 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:44:32.414069 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.404640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34231 (local address 127.0.98.65:43129)
0926 20:44:32.404820 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:32.404861 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:32.405463 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:32.405773 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:32.405777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:32.406423 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:32.406709 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.406715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.407380 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.407381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.408360 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.408362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.408803 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.408804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.408961 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.409441 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:32.409451 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:32.411007 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:32.412633 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:32.412639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:32.412641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:32.412857 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:32.413078 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:32.413079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:32.413080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:32.413184 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:32.413436 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:32.413442 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:32.413594 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 20:44:32.413702 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 20:44:32.413852 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":115,"threads_started":1}
I20260926 20:44:32.414516 10811 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34231
I20260926 20:44:32.414628 10811 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:32.414800 10811 heartbeater.cc:511] Master 127.0.98.126:34231 requested a full tablet report, sending...
I20260926 20:44:32.415297 10620 ts_manager.cc:194] Registered new tserver with Master: c9d670f0a64448df944937aae3cf1cfa (127.0.98.65:33673)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260926 20:44:32.416054 10620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:43129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260926 20:44:32.430886 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34231
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:32.511191 10816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:32.511413 10816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:32.511448 10816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:32.512642 10816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:32.512708 10816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:32.512751 10816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:44:32.514276 10816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:32.514544 10816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:32.514771 10816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:32.519619 10822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:32.519658 10821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:32.519901 10816 server_base.cc:1034] running on GCE node
W20260926 20:44:32.519661 10824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:32.520154 10816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:32.520396 10816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:32.521557 10816 hybrid_clock.cc:648] HybridClock initialized: now 1790455472521544 us; error 29 us; skew 500 ppm
Sep 26 20:44:32 dist-test-slave-4hd3 krb5kdc[10575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:32.523945 10816 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:44:32.524392 10816 webserver.cc:460] Webserver started at http://127.0.98.66:43261/ using document root <none> and password file <none>
I20260926 20:44:32.524619 10816 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:32.524667 10816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:32.524809 10816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:32.525805 10816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "44b80a42822444849b9050434bdf4656"
format_stamp: "Formatted at 2026-09-26 20:44:32 on dist-test-slave-4hd3"
server_key: "2135f427fd063e4a4a5bf5efbfa86573"
server_key_iv: "83a29ffc025dbeebd12385d86e817271"
server_key_version: "encryptionkey@0"
I20260926 20:44:32.526090 10816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "44b80a42822444849b9050434bdf4656"
format_stamp: "Formatted at 2026-09-26 20:44:32 on dist-test-slave-4hd3"
server_key: "2135f427fd063e4a4a5bf5efbfa86573"
server_key_iv: "83a29ffc025dbeebd12385d86e817271"
server_key_version: "encryptionkey@0"
I20260926 20:44:32.527316 10816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:44:32.528069 10831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:32.528299 10816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:44:32.528393 10816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "44b80a42822444849b9050434bdf4656"
format_stamp: "Formatted at 2026-09-26 20:44:32 on dist-test-slave-4hd3"
server_key: "2135f427fd063e4a4a5bf5efbfa86573"
server_key_iv: "83a29ffc025dbeebd12385d86e817271"
server_key_version: "encryptionkey@0"
I20260926 20:44:32.528496 10816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:32.538223 10816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:32.540089 10816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:32.540249 10816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:32.540463 10816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:32.540761 10816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:32.540814 10816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:32.540858 10816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:32.540885 10816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:32.546589 10816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:33189
I20260926 20:44:32.546626 10944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:33189 every 8 connection(s)
I20260926 20:44:32.546962 10816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:44:32 dist-test-slave-4hd3 krb5kdc[10575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:44:32.555389 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 10816
I20260926 20:44:32.555513 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:44:32.555749 393 external_mini_cluster.cc:1428] Setting key 0b1fde0dd72c14606071dfc595824f59
I20260926 20:44:32.556309 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.547651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:46595 (local address 127.0.98.126:34231)
0926 20:44:32.547754 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:32.547756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:32.547944 (+ 188us) server_negotiation.cc:407] Connection header received
0926 20:44:32.548453 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:32.548454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:32.548466 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:32.548523 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:32.549440 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.549940 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.550928 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.551137 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.551220 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.553419 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:32.553421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:32.553421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:32.553429 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:32.554726 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:32.555151 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:32.555153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:32.555153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:32.555218 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:32.555471 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:32.555472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:32.555472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:32.555603 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:32.555735 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:32.556164 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 20:44:32.556246 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:44:32.556445 10947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.547637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34231 (local address 127.0.98.66:46595)
0926 20:44:32.547827 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:32.547862 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:32.548349 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:32.548599 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:32.548603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:32.549153 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:32.549341 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.549346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.550033 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.550034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.550825 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.550826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.551231 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.551231 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.551356 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.551912 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:32.551921 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:32.553315 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:32.554835 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:32.554841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:32.554843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:32.555060 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:32.555286 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:32.555287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:32.555288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:32.555396 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:32.555743 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:32.555749 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:32.555925 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 20:44:32.556043 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 20:44:32.556230 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":108,"threads_started":1}
I20260926 20:44:32.556986 10945 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34231
I20260926 20:44:32.557070 10945 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:32.557261 10945 heartbeater.cc:511] Master 127.0.98.126:34231 requested a full tablet report, sending...
I20260926 20:44:32.557598 10620 ts_manager.cc:194] Registered new tserver with Master: 44b80a42822444849b9050434bdf4656 (127.0.98.66:33189)
I20260926 20:44:32.558109 10620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:46595
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260926 20:44:32.582214 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34231
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:44:32.662886 10950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:44:32.663085 10950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:44:32.663107 10950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:44:32.664224 10950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:44:32.664259 10950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:44:32.664285 10950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:44:32.666531 10950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:44:32.666803 10950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:44:32.667017 10950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:44:32.672024 10958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:32.672067 10955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:44:32.672031 10956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:44:32.672379 10950 server_base.cc:1034] running on GCE node
I20260926 20:44:32.672601 10950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:44:32.672804 10950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:44:32.673995 10950 hybrid_clock.cc:648] HybridClock initialized: now 1790455472673981 us; error 25 us; skew 500 ppm
Sep 26 20:44:32 dist-test-slave-4hd3 krb5kdc[10575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:44:32.676759 10950 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:44:32.677235 10950 webserver.cc:460] Webserver started at http://127.0.98.67:45723/ using document root <none> and password file <none>
I20260926 20:44:32.677492 10950 fs_manager.cc:359] Metadata directory not provided
I20260926 20:44:32.677538 10950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:44:32.677680 10950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:44:32.678699 10950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "bcd3bacc4aaa4afcbdb805e866a8254e"
format_stamp: "Formatted at 2026-09-26 20:44:32 on dist-test-slave-4hd3"
server_key: "e7ee4356990941519aa213a96f0f8450"
server_key_iv: "bd38cf79c60372768cd4474e58f3d9e3"
server_key_version: "encryptionkey@0"
I20260926 20:44:32.679039 10950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "bcd3bacc4aaa4afcbdb805e866a8254e"
format_stamp: "Formatted at 2026-09-26 20:44:32 on dist-test-slave-4hd3"
server_key: "e7ee4356990941519aa213a96f0f8450"
server_key_iv: "bd38cf79c60372768cd4474e58f3d9e3"
server_key_version: "encryptionkey@0"
I20260926 20:44:32.680347 10950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:44:32.681031 10965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:32.681238 10950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:44:32.681308 10950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "bcd3bacc4aaa4afcbdb805e866a8254e"
format_stamp: "Formatted at 2026-09-26 20:44:32 on dist-test-slave-4hd3"
server_key: "e7ee4356990941519aa213a96f0f8450"
server_key_iv: "bd38cf79c60372768cd4474e58f3d9e3"
server_key_version: "encryptionkey@0"
I20260926 20:44:32.681406 10950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:44:32.695953 10950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:44:32.698304 10950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:44:32.698450 10950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:44:32.698693 10950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:44:32.698987 10950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:44:32.699040 10950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:32.699079 10950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:44:32.699105 10950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:44:32.705224 10950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:42949
I20260926 20:44:32.705282 11078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:42949 every 8 connection(s)
I20260926 20:44:32.705611 10950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:44:32.707403 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 10950
I20260926 20:44:32.707487 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:44:32.707707 393 external_mini_cluster.cc:1428] Setting key cdc4697cb3236b7bb08839834525ae7a
Sep 26 20:44:32 dist-test-slave-4hd3 krb5kdc[10575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:44:32.715113 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.706558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:46481 (local address 127.0.98.126:34231)
0926 20:44:32.706625 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:32.706628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:32.706793 (+ 165us) server_negotiation.cc:407] Connection header received
0926 20:44:32.707294 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:32.707296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:32.707309 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:32.707390 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:32.708506 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.709044 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.710137 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.710338 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.710416 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.712581 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:32.712583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:32.712583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:32.712590 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:32.713791 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:32.714190 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:32.714191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:32.714192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:32.714250 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:32.714492 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:32.714493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:32.714493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:32.714609 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:32.714691 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:32.714947 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 20:44:32.715022 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:44:32.715319 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.706517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34231 (local address 127.0.98.67:46481)
0926 20:44:32.706664 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:32.706704 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:32.707157 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:32.707464 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:32.707469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:32.708192 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:44:32.708410 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.708415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.709153 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.709154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.710030 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.710032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.710432 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.710433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.710552 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.711019 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:44:32.711027 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:44:32.712472 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:44:32.713884 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:32.713891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:32.713893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:32.714105 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:32.714314 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:44:32.714315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:44:32.714315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:44:32.714424 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:44:32.714675 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:44:32.714682 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:44:32.714820 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 20:44:32.714927 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 20:44:32.715077 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260926 20:44:32.715816 11079 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34231
I20260926 20:44:32.715903 11079 heartbeater.cc:464] Registering TS with master...
I20260926 20:44:32.716039 11079 heartbeater.cc:511] Master 127.0.98.126:34231 requested a full tablet report, sending...
I20260926 20:44:32.716418 10620 ts_manager.cc:194] Registered new tserver with Master: bcd3bacc4aaa4afcbdb805e866a8254e (127.0.98.67:42949)
I20260926 20:44:32.716784 10620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:46481
I20260926 20:44:32.719323 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:44:32.726974 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.720617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49598 (local address 127.0.98.126:34231)
0926 20:44:32.720677 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:32.720679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:32.720782 (+ 103us) server_negotiation.cc:407] Connection header received
0926 20:44:32.720843 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:32.720844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:32.720852 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:32.720904 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:44:32.721413 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.721896 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.722589 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.722787 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.722846 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.724218 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:44:32.724219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:44:32.724219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:44:32.724225 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:44:32.725679 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:32.726130 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:32.726131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:32.726131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:32.726193 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:44:32.726391 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:44:32.726392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:44:32.726392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:44:32.726538 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:44:32.726640 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:32.726787 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 20:44:32.726871 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:44:32.728645 10620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:44:32.730376 10620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:44:32.736812 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.732618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:42949 (local address 127.0.0.1:45874)
0926 20:44:32.732748 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:32.732791 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:32.732877 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:32.733393 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:32.733396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:32.733398 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:32.733465 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.733467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.734833 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.734834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.735702 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.735704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.736525 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.736527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.736601 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.736614 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 20:44:32.736688 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:44:32.736735 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260926 20:44:32.736868 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.732801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45874 (local address 127.0.98.67:42949)
0926 20:44:32.733234 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:32.733237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:32.733244 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:44:32.733275 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:32.733276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:32.733290 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:32.733374 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:32.733554 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.734708 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.735820 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.736327 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.736429 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.736491 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:32.736727 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 20:44:32.736772 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":75,"threads_started":1}
I20260926 20:44:32.737377 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.732830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:33189 (local address 127.0.0.1:48990)
0926 20:44:32.733466 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:32.733477 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:32.733532 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:32.733857 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:32.733858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:32.733860 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:32.733895 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.733896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.734725 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.734726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.736058 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.736060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.737147 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.737148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.737217 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.737221 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:44:32.737267 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 20:44:32.737308 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":76,"thread_start_us":44,"threads_started":1}
I20260926 20:44:32.737665 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.732618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33673 (local address 127.0.0.1:48678)
0926 20:44:32.733148 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:32.733157 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:32.733224 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:32.733565 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:32.733566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:32.733568 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:32.733624 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.733626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.734868 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.734868 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.736310 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.736312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.737449 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.737450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.737526 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.737530 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:44:32.737559 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 20:44:32.737597 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":43,"spinlock_wait_cycles":896,"thread_start_us":75,"threads_started":1}
I20260926 20:44:32.737686 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.732933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.0.98.66:33189)
0926 20:44:32.733093 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:32.733098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:32.733533 (+ 435us) server_negotiation.cc:407] Connection header received
0926 20:44:32.733752 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:32.733757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:32.733775 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:32.733860 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:32.733941 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.734623 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.736161 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.736810 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.736875 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.737527 (+ 652us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:32.737557 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 20:44:32.737609 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260926 20:44:32.737561 11013 tablet_service.cc:1467] Processing CreateTablet for tablet c9fb2986844145aebca4a138bcaa90b7 (DEFAULT_TABLE table=test-table [id=c86db51170664a70a1256baa46b1fbb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:32.737965 11013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fb2986844145aebca4a138bcaa90b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:32.738222 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.732982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48678 (local address 127.0.98.65:33673)
0926 20:44:32.733137 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:32.733142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:32.733211 (+ 69us) server_negotiation.cc:407] Connection header received
0926 20:44:32.733303 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:32.733306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:32.733320 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:32.733419 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:32.733707 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.734778 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.736398 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.737343 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.737635 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.737701 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:32.738100 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 20:44:32.738144 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260926 20:44:32.738359 10879 tablet_service.cc:1467] Processing CreateTablet for tablet c9fb2986844145aebca4a138bcaa90b7 (DEFAULT_TABLE table=test-table [id=c86db51170664a70a1256baa46b1fbb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:32.738610 10879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fb2986844145aebca4a138bcaa90b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:32.738847 10745 tablet_service.cc:1467] Processing CreateTablet for tablet c9fb2986844145aebca4a138bcaa90b7 (DEFAULT_TABLE table=test-table [id=c86db51170664a70a1256baa46b1fbb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:44:32.739143 10745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fb2986844145aebca4a138bcaa90b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:44:32.741171 11093 tablet_bootstrap.cc:492] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656: Bootstrap starting.
I20260926 20:44:32.741765 11093 tablet_bootstrap.cc:654] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:32.742175 11093 log.cc:824] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656: Log is configured to *not* fsync() on all Append() calls
I20260926 20:44:32.742333 11095 tablet_bootstrap.cc:492] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa: Bootstrap starting.
I20260926 20:44:32.742857 11094 tablet_bootstrap.cc:492] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e: Bootstrap starting.
I20260926 20:44:32.743003 11095 tablet_bootstrap.cc:654] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:32.743021 11093 tablet_bootstrap.cc:492] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656: No bootstrap required, opened a new log
I20260926 20:44:32.743100 11093 ts_tablet_manager.cc:1397] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:44:32.743587 11095 log.cc:824] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa: Log is configured to *not* fsync() on all Append() calls
I20260926 20:44:32.743664 11094 tablet_bootstrap.cc:654] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e: Neither blocks nor log segments found. Creating new log.
I20260926 20:44:32.744268 11094 log.cc:824] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:44:32.744632 11095 tablet_bootstrap.cc:492] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa: No bootstrap required, opened a new log
I20260926 20:44:32.744721 11095 ts_tablet_manager.cc:1397] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 20:44:32.745059 11094 tablet_bootstrap.cc:492] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e: No bootstrap required, opened a new log
I20260926 20:44:32.745157 11094 ts_tablet_manager.cc:1397] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:44:32.745072 11093 raft_consensus.cc:348] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd3bacc4aaa4afcbdb805e866a8254e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42949 } } peers { permanent_uuid: "c9d670f0a64448df944937aae3cf1cfa" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33673 } } peers { permanent_uuid: "44b80a42822444849b9050434bdf4656" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33189 } }
I20260926 20:44:32.745289 11093 raft_consensus.cc:374] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:32.745337 11093 raft_consensus.cc:729] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44b80a42822444849b9050434bdf4656, State: Initialized, Role: FOLLOWER
I20260926 20:44:32.745451 11093 consensus_queue.cc:260] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd3bacc4aaa4afcbdb805e866a8254e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42949 } } peers { permanent_uuid: "c9d670f0a64448df944937aae3cf1cfa" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33673 } } peers { permanent_uuid: "44b80a42822444849b9050434bdf4656" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33189 } }
I20260926 20:44:32.745743 11093 ts_tablet_manager.cc:1428] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:44:32.745872 10945 heartbeater.cc:503] Master 127.0.98.126:34231 was elected leader, sending a full tablet report...
I20260926 20:44:32.747123 11094 raft_consensus.cc:348] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd3bacc4aaa4afcbdb805e866a8254e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42949 } } peers { permanent_uuid: "c9d670f0a64448df944937aae3cf1cfa" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33673 } } peers { permanent_uuid: "44b80a42822444849b9050434bdf4656" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33189 } }
I20260926 20:44:32.747123 11095 raft_consensus.cc:348] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd3bacc4aaa4afcbdb805e866a8254e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42949 } } peers { permanent_uuid: "c9d670f0a64448df944937aae3cf1cfa" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33673 } } peers { permanent_uuid: "44b80a42822444849b9050434bdf4656" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33189 } }
I20260926 20:44:32.747308 11095 raft_consensus.cc:374] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:32.747308 11094 raft_consensus.cc:374] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:44:32.747345 11095 raft_consensus.cc:729] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d670f0a64448df944937aae3cf1cfa, State: Initialized, Role: FOLLOWER
I20260926 20:44:32.747345 11094 raft_consensus.cc:729] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcd3bacc4aaa4afcbdb805e866a8254e, State: Initialized, Role: FOLLOWER
I20260926 20:44:32.747463 11095 consensus_queue.cc:260] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd3bacc4aaa4afcbdb805e866a8254e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42949 } } peers { permanent_uuid: "c9d670f0a64448df944937aae3cf1cfa" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33673 } } peers { permanent_uuid: "44b80a42822444849b9050434bdf4656" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33189 } }
I20260926 20:44:32.747463 11094 consensus_queue.cc:260] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd3bacc4aaa4afcbdb805e866a8254e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42949 } } peers { permanent_uuid: "c9d670f0a64448df944937aae3cf1cfa" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33673 } } peers { permanent_uuid: "44b80a42822444849b9050434bdf4656" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33189 } }
I20260926 20:44:32.747804 11079 heartbeater.cc:503] Master 127.0.98.126:34231 was elected leader, sending a full tablet report...
I20260926 20:44:32.747911 11095 ts_tablet_manager.cc:1428] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 20:44:32.748044 11094 ts_tablet_manager.cc:1428] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:44:32.748070 10811 heartbeater.cc:503] Master 127.0.98.126:34231 was elected leader, sending a full tablet report...
W20260926 20:44:32.797497 10946 tablet.cc:2369] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:44:32.904729 10812 tablet.cc:2369] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:44:32.915064 11099 raft_consensus.cc:482] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:44:32.915212 11099 raft_consensus.cc:504] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd3bacc4aaa4afcbdb805e866a8254e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42949 } } peers { permanent_uuid: "c9d670f0a64448df944937aae3cf1cfa" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33673 } } peers { permanent_uuid: "44b80a42822444849b9050434bdf4656" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33189 } }
I20260926 20:44:32.915598 11099 leader_election.cc:288] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcd3bacc4aaa4afcbdb805e866a8254e (127.0.98.67:42949), c9d670f0a64448df944937aae3cf1cfa (127.0.98.65:33673)
I20260926 20:44:32.918584 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.915843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:44931 (local address 127.0.98.65:33673)
0926 20:44:32.915888 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:32.915891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:32.915920 (+ 29us) server_negotiation.cc:407] Connection header received
0926 20:44:32.916001 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:32.916002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:32.916014 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:32.916068 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:32.916301 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.916852 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.917817 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.918224 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.918309 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.918320 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:32.918482 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 20:44:32.918517 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:44:32.918587 10947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.915806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33673 (local address 127.0.98.66:44931)
0926 20:44:32.915871 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:32.915883 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:32.915937 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:32.916135 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:32.916136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:32.916138 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:32.916216 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.916219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.916935 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.916936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.917725 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.917727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.918310 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.918311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.918397 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.918400 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:44:32.918474 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:44:32.918516 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":38}
I20260926 20:44:32.918661 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.916090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:46661 (local address 127.0.98.67:42949)
0926 20:44:32.916160 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:32.916162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:32.916170 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:44:32.916199 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:32.916199 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:32.916207 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:32.916271 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:32.916549 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.916999 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.917929 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.918312 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.918388 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.918398 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:32.918556 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 20:44:32.918595 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:44:32.918948 10765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fb2986844145aebca4a138bcaa90b7" candidate_uuid: "44b80a42822444849b9050434bdf4656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9d670f0a64448df944937aae3cf1cfa" is_pre_election: true
I20260926 20:44:32.919086 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.915814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:42949 (local address 127.0.98.66:46661)
0926 20:44:32.916010 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:32.916022 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:32.916080 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:32.916374 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:32.916374 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:32.916376 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:32.916445 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.916447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.917078 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.917079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.917828 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.917830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.918401 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.918402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.918473 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.918476 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:44:32.918548 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 20:44:32.918589 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":4,"thread_start_us":140,"threads_started":1}
I20260926 20:44:32.919106 10765 raft_consensus.cc:2454] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44b80a42822444849b9050434bdf4656 in term 0.
I20260926 20:44:32.919160 11101 raft_consensus.cc:482] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:44:32.919265 11101 raft_consensus.cc:504] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd3bacc4aaa4afcbdb805e866a8254e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42949 } } peers { permanent_uuid: "c9d670f0a64448df944937aae3cf1cfa" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33673 } } peers { permanent_uuid: "44b80a42822444849b9050434bdf4656" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33189 } }
I20260926 20:44:32.919371 10833 leader_election.cc:302] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44b80a42822444849b9050434bdf4656, c9d670f0a64448df944937aae3cf1cfa; no voters:
I20260926 20:44:32.919471 11033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fb2986844145aebca4a138bcaa90b7" candidate_uuid: "44b80a42822444849b9050434bdf4656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcd3bacc4aaa4afcbdb805e866a8254e" is_pre_election: true
I20260926 20:44:32.919517 11099 raft_consensus.cc:2781] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:44:32.919629 11033 raft_consensus.cc:2454] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44b80a42822444849b9050434bdf4656 in term 0.
I20260926 20:44:32.919776 11099 raft_consensus.cc:482] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:44:32.919863 11099 raft_consensus.cc:3037] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:32.919924 11101 leader_election.cc:288] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcd3bacc4aaa4afcbdb805e866a8254e (127.0.98.67:42949), 44b80a42822444849b9050434bdf4656 (127.0.98.66:33189)
I20260926 20:44:32.920652 11099 raft_consensus.cc:504] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd3bacc4aaa4afcbdb805e866a8254e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42949 } } peers { permanent_uuid: "c9d670f0a64448df944937aae3cf1cfa" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33673 } } peers { permanent_uuid: "44b80a42822444849b9050434bdf4656" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33189 } }
I20260926 20:44:32.920828 11099 leader_election.cc:288] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [CANDIDATE]: Term 1 election: Requested vote from peers bcd3bacc4aaa4afcbdb805e866a8254e (127.0.98.67:42949), c9d670f0a64448df944937aae3cf1cfa (127.0.98.65:33673)
I20260926 20:44:32.920971 11033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fb2986844145aebca4a138bcaa90b7" candidate_uuid: "44b80a42822444849b9050434bdf4656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcd3bacc4aaa4afcbdb805e866a8254e"
I20260926 20:44:32.921049 11033 raft_consensus.cc:3037] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:32.921876 11033 raft_consensus.cc:2454] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44b80a42822444849b9050434bdf4656 in term 1.
I20260926 20:44:32.922055 10834 leader_election.cc:302] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44b80a42822444849b9050434bdf4656, bcd3bacc4aaa4afcbdb805e866a8254e; no voters:
I20260926 20:44:32.922075 10765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fb2986844145aebca4a138bcaa90b7" candidate_uuid: "44b80a42822444849b9050434bdf4656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9d670f0a64448df944937aae3cf1cfa"
I20260926 20:44:32.922140 10765 raft_consensus.cc:3037] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:44:32.922189 11099 raft_consensus.cc:2781] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:44:32.922411 11099 raft_consensus.cc:686] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [term 1 LEADER]: Becoming Leader. State: Replica: 44b80a42822444849b9050434bdf4656, State: Running, Role: LEADER
I20260926 20:44:32.922559 11099 consensus_queue.cc:237] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd3bacc4aaa4afcbdb805e866a8254e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42949 } } peers { permanent_uuid: "c9d670f0a64448df944937aae3cf1cfa" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33673 } } peers { permanent_uuid: "44b80a42822444849b9050434bdf4656" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33189 } }
I20260926 20:44:32.923075 10765 raft_consensus.cc:2454] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44b80a42822444849b9050434bdf4656 in term 1.
I20260926 20:44:32.923547 10619 catalog_manager.cc:5495] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44b80a42822444849b9050434bdf4656 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "44b80a42822444849b9050434bdf4656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd3bacc4aaa4afcbdb805e866a8254e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9d670f0a64448df944937aae3cf1cfa" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44b80a42822444849b9050434bdf4656" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33189 } health_report { overall_health: HEALTHY } } }
I20260926 20:44:32.924400 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.919805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:42949 (local address 127.0.98.65:35901)
0926 20:44:32.921110 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:32.921119 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:32.921185 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:32.921349 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:32.921350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:32.921351 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:32.921428 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.921429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.922240 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.922241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.923383 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.923384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.924135 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.924136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.924225 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.924228 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:44:32.924307 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:44:32.924332 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1293,"thread_start_us":56,"threads_started":1}
I20260926 20:44:32.924405 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.919801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:33189 (local address 127.0.98.65:53379)
0926 20:44:32.920917 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:32.920938 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:44:32.921024 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:44:32.921219 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:44:32.921220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:44:32.921222 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:44:32.921266 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.921267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.922113 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.922114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.923377 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:44:32.923379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.924138 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.924139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:44:32.924205 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.924209 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:44:32.924269 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 20:44:32.924314 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"mutex_wait_us":3,"thread_start_us":63,"threads_started":1}
I20260926 20:44:32.924467 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.919937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:35901 (local address 127.0.98.67:42949)
0926 20:44:32.920001 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:32.920002 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:32.921162 (+ 1160us) server_negotiation.cc:407] Connection header received
0926 20:44:32.921255 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:32.921256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:32.921265 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:32.921294 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:32.921496 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.922143 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.923480 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.924018 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.924064 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.924074 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:32.924361 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 20:44:32.924397 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:44:32.924623 11033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fb2986844145aebca4a138bcaa90b7" candidate_uuid: "c9d670f0a64448df944937aae3cf1cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcd3bacc4aaa4afcbdb805e866a8254e" is_pre_election: true
I20260926 20:44:32.924708 11033 raft_consensus.cc:2379] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9d670f0a64448df944937aae3cf1cfa in current term 1: Already voted for candidate 44b80a42822444849b9050434bdf4656 in this term.
I20260926 20:44:32.924712 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.919937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:53379 (local address 127.0.98.66:33189)
0926 20:44:32.919974 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:32.919975 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:32.921023 (+ 1048us) server_negotiation.cc:407] Connection header received
0926 20:44:32.921075 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:32.921076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:32.921087 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:32.921139 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:44:32.921349 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.922029 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.923481 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.924043 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.924081 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.924089 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:32.924590 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 20:44:32.924631 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:44:32.924923 10899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fb2986844145aebca4a138bcaa90b7" candidate_uuid: "c9d670f0a64448df944937aae3cf1cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44b80a42822444849b9050434bdf4656" is_pre_election: true
I20260926 20:44:32.925192 10699 leader_election.cc:302] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9d670f0a64448df944937aae3cf1cfa; no voters: 44b80a42822444849b9050434bdf4656, bcd3bacc4aaa4afcbdb805e866a8254e
I20260926 20:44:32.925315 11101 raft_consensus.cc:2726] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 20:44:32.956318 11080 tablet.cc:2369] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:44:32.965968 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.963423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48996 (local address 127.0.98.66:33189)
0926 20:44:32.963490 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 20:44:32.963492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:44:32.963502 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:44:32.963546 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:44:32.963552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:44:32.963570 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:44:32.963650 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:44:32.963845 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.964346 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.965118 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:44:32.965317 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:44:32.965392 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:44:32.965584 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:44:32.965693 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:44:32.965794 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 20:44:32.965837 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 20:44:32.965858 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:44:32.967926 10765 raft_consensus.cc:1261] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa [term 1 FOLLOWER]: Refusing update from remote peer 44b80a42822444849b9050434bdf4656: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:44:32.967932 11033 raft_consensus.cc:1261] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e [term 1 FOLLOWER]: Refusing update from remote peer 44b80a42822444849b9050434bdf4656: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:44:32.968267 11105 consensus_queue.cc:1035] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9d670f0a64448df944937aae3cf1cfa" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:44:32.968530 11105 consensus_queue.cc:1035] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3bacc4aaa4afcbdb805e866a8254e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:44:32.971057 11110 mvcc.cc:205] Tried to move back new op lower bound from 7333705617273171968 to 7333705617091768320. Current Snapshot: MvccSnapshot[applied={T|T < 7333705617273171968}]
I20260926 20:44:32.971122 11109 mvcc.cc:205] Tried to move back new op lower bound from 7333705617273171968 to 7333705617091768320. Current Snapshot: MvccSnapshot[applied={T|T < 7333705617273171968}]
I20260926 20:44:32.973330 10620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:44:32.973417 10620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:44:32.974148 10620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55a9a6b50fcf42fba376d35e61841f62: Sending DeleteTablet for 3 replicas of tablet c9fb2986844145aebca4a138bcaa90b7
I20260926 20:44:32.974432 11013 tablet_service.cc:1514] Processing DeleteTablet for tablet c9fb2986844145aebca4a138bcaa90b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:32 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:45874
I20260926 20:44:32.974471 10879 tablet_service.cc:1514] Processing DeleteTablet for tablet c9fb2986844145aebca4a138bcaa90b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:32 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:48990
I20260926 20:44:32.974678 11118 tablet_replica.cc:330] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e: stopping tablet replica
I20260926 20:44:32.974757 10745 tablet_service.cc:1514] Processing DeleteTablet for tablet c9fb2986844145aebca4a138bcaa90b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:44:32 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:48678
I20260926 20:44:32.974787 11118 raft_consensus.cc:2229] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:44:32.974893 11118 raft_consensus.cc:2258] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:32.975013 11120 tablet_replica.cc:330] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa: stopping tablet replica
I20260926 20:44:32.975142 11120 raft_consensus.cc:2229] T c9fb2986844145aebca4a138bcaa90b7 P c9d670f0a64448df944937aae3cf1cfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:44:32.975167 11119 tablet_replica.cc:330] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656: stopping tablet replica
I20260926 20:44:32.975191 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 10682
I20260926 20:44:32.975258 11119 raft_consensus.cc:2229] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:44:32.975374 11119 raft_consensus.cc:2258] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:44:32.976658 11119 ts_tablet_manager.cc:1905] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:32.978025 11118 ts_tablet_manager.cc:1905] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:44:32.978497 11119 ts_tablet_manager.cc:1918] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:44:32.978574 11119 log.cc:1196] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/c9fb2986844145aebca4a138bcaa90b7
I20260926 20:44:32.978847 11119 ts_tablet_manager.cc:1939] T c9fb2986844145aebca4a138bcaa90b7 P 44b80a42822444849b9050434bdf4656: Deleting consensus metadata
I20260926 20:44:32.979256 10605 catalog_manager.cc:4841] TS 44b80a42822444849b9050434bdf4656 (127.0.98.66:33189): tablet c9fb2986844145aebca4a138bcaa90b7 (table test-table [id=c86db51170664a70a1256baa46b1fbb3]) successfully deleted
I20260926 20:44:32.979705 11118 ts_tablet_manager.cc:1918] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:44:32.979810 11118 log.cc:1196] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/c9fb2986844145aebca4a138bcaa90b7
I20260926 20:44:32.980070 11118 ts_tablet_manager.cc:1939] T c9fb2986844145aebca4a138bcaa90b7 P bcd3bacc4aaa4afcbdb805e866a8254e: Deleting consensus metadata
I20260926 20:44:32.980420 10606 catalog_manager.cc:4841] TS bcd3bacc4aaa4afcbdb805e866a8254e (127.0.98.67:42949): tablet c9fb2986844145aebca4a138bcaa90b7 (table test-table [id=c86db51170664a70a1256baa46b1fbb3]) successfully deleted
W20260926 20:44:32.982312 10605 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:33673 (error 108)
I20260926 20:44:32.982537 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 10816
I20260926 20:44:32.983079 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:44:32.982615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33673 (local address 127.0.0.1:48690)
0926 20:44:32.983008 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0926 20:44:32.983041 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:33673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376}
W20260926 20:44:32.983347 10605 catalog_manager.cc:4584] TS c9d670f0a64448df944937aae3cf1cfa (127.0.98.65:33673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9fb2986844145aebca4a138bcaa90b7: Network error: Client connection negotiation failed: client connection to 127.0.98.65:33673: connect: Connection refused (error 111)
I20260926 20:44:32.986919 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 10950
I20260926 20:44:32.990983 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 10588
2026-09-26T20:44:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2082 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:44:33 dist-test-slave-4hd3 krb5kdc[11124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:44:33 dist-test-slave-4hd3 krb5kdc[11124](info): set up 2 sockets
Sep 26 20:44:33 dist-test-slave-4hd3 krb5kdc[11124](info): commencing operation
krb5kdc: starting...
W20260926 20:44:34.016641 393 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:44:34 dist-test-slave-4hd3 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:44:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:44:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/rangeradmin_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/rangeradmin_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/rangerlookup_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/rangerlookup_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
I20260926 20:44:34.182427 393 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDRmUyo/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/postgres -l logfile start
2026-09-26 20:44:36.638 UTC [11154] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 20:44:36.638 UTC [11154] LOG: listening on IPv4 address "127.0.98.84", port 56145
2026-09-26 20:44:36.642 UTC [11154] LOG: listening on Unix socket "/tmp/.s.PGSQL.56145"
2026-09-26 20:44:36.676 UTC [11157] LOG: database system was shut down at 2026-09-26 20:44:35 UTC
2026-09-26 20:44:36.679 UTC [11154] LOG: database system is ready to accept connections
I20260926 20:44:37.621089 393 mini_postgres.cc:96] Postgres bound to 56145
2026-09-26 20:44:37.627 UTC [11165] FATAL: database "slave" does not exist
127.0.98.84:56145 - accepting connections
I20260926 20:44:37.627970 393 mini_ranger.cc:161] Starting Ranger...
I20260926 20:44:37.642315 393 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 20:44:38.095822 393 mini_ranger.cc:87] Created ranger Postgres database
I20260926 20:44:38.095906 393 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ranger-admin
2026-09-26 20:44:38,366 [I] DB FLAVOR :POSTGRES
2026-09-26 20:44:38,366 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:44:38,366 [I] Checking connection..
2026-09-26 20:44:38,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 20:44:38,631 [I] Checking connection passed.
2026-09-26 20:44:38,631 [I] --------- Verifying version history table ---------
2026-09-26 20:44:38,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:44:38,873 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:44:38,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:44:39,136 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:44:39,136 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 20:44:39,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 20:44:39.377 UTC [11229] WARNING: there is no transaction in progress
2026-09-26 20:44:39,383 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 20:44:39,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:44:39,633 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 20:44:39,633 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:44:39,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 20:44:39,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 20:44:40,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4hd3.c.gcp-upstream.internal','N') ;"
2026-09-26 20:44:40,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:44:40,601 [I] Table x_portal_user does not exist in database ranger
2026-09-26 20:44:40,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:44:40,854 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 20:44:40,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:44:41,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:44:41,353 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 20:44:41,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 20:44:42.300 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 20:44:42.316 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 20:44:42.333 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 20:44:42.349 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 20:44:42.365 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 20:44:42.383 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 20:44:42.438 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 20:44:42.444 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 20:44:42.458 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 20:44:42.469 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 20:44:42.480 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 20:44:42.494 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 20:44:42.503 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 20:44:42.509 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 20:44:42.515 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 20:44:42.521 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 20:44:43,161 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 20:44:43,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 20:44:43,383 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 20:44:43,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:44:43,619 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 20:44:43,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:44:43,876 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 20:44:43,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:44:44,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:44:44,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4hd3.c.gcp-upstream.internal';"
2026-09-26 20:44:44,589 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 20:44:44,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 20:44:44,821 [I] DB_PATCHES have already been applied
I20260926 20:44:44.827531 393 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDRmUyo/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 20:44:44.827623 393 mini_ranger.cc:193] Using host: 127.0.98.84
I20260926 20:44:44.830586 393 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.84:35749
Sep 26, 2026 8:44:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:44:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:44:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 8:44:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 8:44:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 8:44:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 8:44:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 20:44:45.942541 393 mini_ranger.cc:160] Time spent starting Ranger: real 8.315s user 0.000s sys 0.009s
Sep 26, 2026 8:45:05 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 8:45:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 8:45:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 8:45:06 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 8:45:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 8:45:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 8:45:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 8:45:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 8:45:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 8:45:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 8:45:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 8:45:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 8:45:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 8:45:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 8:45:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 8:45:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 8:45:06 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 8:45:06 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 20:45:07.508738 393 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/rangerkms_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/rangerkms_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 20:45:07 dist-test-slave-4hd3 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455507, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 20:45:07.590977 393 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 20:45:07.600365 393 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 20:45:08.417580 393 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 20:45:08.417675 393 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ranger-kms
I20260926 20:45:08.417695 393 mini_ranger_kms.cc:227] Using postgres at 127.0.98.84:56145
2026-09-26 20:45:08,589 [I] DB FLAVOR :POSTGRES
2026-09-26 20:45:08,589 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:45:08,589 [I] Checking connection
2026-09-26 20:45:08,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 20:45:08,850 [I] connection success
2026-09-26 20:45:08,850 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 20:45:08,850 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 20:45:08,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 20:45:09,109 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 20:45:09,110 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:45:09,110 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 20:45:09,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:56145/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 20:45:09,360 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 20:45:09.820223 393 mini_ranger_kms.cc:310] Created kms service
I20260926 20:45:09.992856 393 mini_ranger_kms.cc:326] Created kudu user
I20260926 20:45:10.036868 393 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 20:45:10 dist-test-slave-4hd3 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455507, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260926 20:45:10.422263 393 mini_ranger_kms.cc:384] Added ranger policy
I20260926 20:45:10.422372 393 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 20:45:10.422394 393 mini_ranger_kms.cc:240] Using host: 127.0.98.84
I20260926 20:45:10.425938 393 mini_ranger_kms.cc:276] Ranger KMS PID: 11517
I20260926 20:45:10.426051 393 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.84:54443
Sep 26, 2026 8:45:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:45:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:45:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 8:45:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 8:45:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 8:45:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 20:45:12.770435 393 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.179s user 0.009s sys 0.146s
I20260926 20:45:12.770533 393 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 20:45:12.770607 393 mini_ranger_kms.cc:401] 127.0.98.84:54443/kms/v1/keys
Sep 26 20:45:13 dist-test-slave-4hd3 krb5kdc[11124](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790455513, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 20:45:13 dist-test-slave-4hd3 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455513, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.84@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
Sep 26 20:45:13 dist-test-slave-4hd3 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455513, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.84@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
Sep 26 20:45:15 dist-test-slave-4hd3 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:45:16.021070 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38257
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:38257
--ranger_config_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:54443/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:45:16.141880 11568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:45:16.142179 11568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:45:16.142210 11568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:45:16.143916 11568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:45:16.143972 11568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:45:16.143992 11568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:45:16.144011 11568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:45:16.144027 11568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:45:16.146234 11568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.84:54443/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:38257
--ranger_config_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:45:16.146492 11568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:45:16.146728 11568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:45:16.152010 11576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:45:16.152016 11574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:45:16.152161 11568 server_base.cc:1034] running on GCE node
W20260926 20:45:16.152235 11573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:45:16.152475 11568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:45:16.152804 11568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:45:16.153968 11568 hybrid_clock.cc:648] HybridClock initialized: now 1790455516153948 us; error 37 us; skew 500 ppm
Sep 26 20:45:16 dist-test-slave-4hd3 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:45:16.156621 11568 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:45:16.157215 11568 webserver.cc:460] Webserver started at http://127.0.98.126:33223/ using document root <none> and password file <none>
I20260926 20:45:16.157501 11568 fs_manager.cc:359] Metadata directory not provided
I20260926 20:45:16.157547 11568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:45:16.157673 11568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:45:16 dist-test-slave-4hd3 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260926 20:45:16.204241 11568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "495255bba4cb437ea28c64c42e6c106c"
format_stamp: "Formatted at 2026-09-26 20:45:16 on dist-test-slave-4hd3"
server_key: "058c1ea426b8eb0e3e342a193ad8da32"
server_key_iv: "ebcb7ace3d095f62fdaa49b6146f8778"
server_key_version: "kuduclusterkey@0"
I20260926 20:45:16.204618 11568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "495255bba4cb437ea28c64c42e6c106c"
format_stamp: "Formatted at 2026-09-26 20:45:16 on dist-test-slave-4hd3"
server_key: "058c1ea426b8eb0e3e342a193ad8da32"
server_key_iv: "ebcb7ace3d095f62fdaa49b6146f8778"
server_key_version: "kuduclusterkey@0"
I20260926 20:45:16.206039 11568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:45:16.217361 11583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:45:16.217558 11568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:45:16.217664 11568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "495255bba4cb437ea28c64c42e6c106c"
format_stamp: "Formatted at 2026-09-26 20:45:16 on dist-test-slave-4hd3"
server_key: "058c1ea426b8eb0e3e342a193ad8da32"
server_key_iv: "ebcb7ace3d095f62fdaa49b6146f8778"
server_key_version: "kuduclusterkey@0"
I20260926 20:45:16.217784 11568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:45:16.240612 11568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:45:16.242841 11568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:45:16.243003 11568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:45:16.248667 11568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:38257
I20260926 20:45:16.248852 11635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:38257 every 8 connection(s)
I20260926 20:45:16.249198 11568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:45:16.249897 11636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:45:16.252421 11636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c: Bootstrap starting.
I20260926 20:45:16.253419 11636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c: Neither blocks nor log segments found. Creating new log.
I20260926 20:45:16.253686 11636 log.cc:824] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:45:16.254377 11636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c: No bootstrap required, opened a new log
I20260926 20:45:16.256541 11636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495255bba4cb437ea28c64c42e6c106c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38257 } }
I20260926 20:45:16.256722 11636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:45:16.256771 11636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495255bba4cb437ea28c64c42e6c106c, State: Initialized, Role: FOLLOWER
I20260926 20:45:16.256946 11636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495255bba4cb437ea28c64c42e6c106c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38257 } }
I20260926 20:45:16.257047 11636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:45:16.257091 11636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:45:16.257138 11636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:45:16.257671 11636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495255bba4cb437ea28c64c42e6c106c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38257 } }
I20260926 20:45:16.257791 11636 leader_election.cc:302] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 495255bba4cb437ea28c64c42e6c106c; no voters:
I20260926 20:45:16.257938 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 11568
I20260926 20:45:16.257954 11636 leader_election.cc:288] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:45:16.258002 11639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:45:16.258078 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:45:16.258131 11639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c [term 1 LEADER]: Becoming Leader. State: Replica: 495255bba4cb437ea28c64c42e6c106c, State: Running, Role: LEADER
I20260926 20:45:16.258204 11639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495255bba4cb437ea28c64c42e6c106c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38257 } }
I20260926 20:45:16.258268 11636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:45:16.258279 393 external_mini_cluster.cc:1428] Setting key 2fa6348e0c92c124141e003310f2f018
I20260926 20:45:16.258991 11639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "495255bba4cb437ea28c64c42e6c106c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495255bba4cb437ea28c64c42e6c106c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38257 } } }
I20260926 20:45:16.259177 11639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c [sys.catalog]: This master's current role is: LEADER
I20260926 20:45:16.259013 11640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 495255bba4cb437ea28c64c42e6c106c. Latest consensus state: current_term: 1 leader_uuid: "495255bba4cb437ea28c64c42e6c106c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495255bba4cb437ea28c64c42e6c106c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38257 } } }
I20260926 20:45:16.259426 11640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c [sys.catalog]: This master's current role is: LEADER
I20260926 20:45:16.259591 11650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:45:16.259811 11636 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 20:45:16.260154 11650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 20:45:16 dist-test-slave-4hd3 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:45:16.271963 11659 mvcc.cc:205] Tried to move back new op lower bound from 7333705794637733888 to 7333705794593873920. Current Snapshot: MvccSnapshot[applied={T|T < 7333705794637733888}]
I20260926 20:45:16.274430 11650 catalog_manager.cc:1347] Generated new cluster ID: 8e6de430933741d5a33a1857884cc0fc
I20260926 20:45:16.274495 11650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:45:16.274842 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:16.258759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53600 (local address 127.0.98.126:38257)
0926 20:45:16.258932 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 20:45:16.258942 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:45:16.259332 (+ 390us) server_negotiation.cc:407] Connection header received
0926 20:45:16.259547 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:45:16.259555 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:45:16.259616 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:45:16.259757 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:45:16.261759 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:16.264299 (+ 2540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:16.265504 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:16.266150 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:16.266240 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:45:16.270004 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:45:16.270009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:45:16.270009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:45:16.270021 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:45:16.273038 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:45:16.273478 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:45:16.273481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:45:16.273482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:45:16.273586 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:45:16.273805 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:45:16.273806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:45:16.273806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:45:16.274159 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:45:16.274275 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:45:16.274560 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 20:45:16.274691 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260926 20:45:16.290465 11650 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:45:16.291230 11650 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:45:16.302659 11650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c: Generated new TSK 0
I20260926 20:45:16.302934 11650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260926 20:45:16.963089 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:54443/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38257
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:45:17.071440 11681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:45:17.071749 11681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:45:17.071787 11681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:45:17.073310 11681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:45:17.073379 11681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:45:17.073427 11681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:45:17.074987 11681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.84:54443/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:45:17.075254 11681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:45:17.075492 11681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:45:17.080812 11689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:45:17.080824 11686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:45:17.081207 11687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:45:17.081122 11681 server_base.cc:1034] running on GCE node
I20260926 20:45:17.081477 11681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:45:17.081763 11681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:45:17.082934 11681 hybrid_clock.cc:648] HybridClock initialized: now 1790455517082921 us; error 27 us; skew 500 ppm
Sep 26 20:45:17 dist-test-slave-4hd3 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:45:17.086031 11681 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:45:17.086450 11681 webserver.cc:460] Webserver started at http://127.0.98.65:33235/ using document root <none> and password file <none>
I20260926 20:45:17.086701 11681 fs_manager.cc:359] Metadata directory not provided
I20260926 20:45:17.086776 11681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:45:17.086926 11681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:45:17 dist-test-slave-4hd3 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260926 20:45:17.101449 11681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "4508a416630345649157e09dd4be2a33"
format_stamp: "Formatted at 2026-09-26 20:45:17 on dist-test-slave-4hd3"
server_key: "0376af3a4edc37139b122fde7bcccf5b"
server_key_iv: "5418eb3f779f841c68fbbdcafc024fac"
server_key_version: "kuduclusterkey@0"
I20260926 20:45:17.101871 11681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "4508a416630345649157e09dd4be2a33"
format_stamp: "Formatted at 2026-09-26 20:45:17 on dist-test-slave-4hd3"
server_key: "0376af3a4edc37139b122fde7bcccf5b"
server_key_iv: "5418eb3f779f841c68fbbdcafc024fac"
server_key_version: "kuduclusterkey@0"
I20260926 20:45:17.103683 11681 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260926 20:45:17.119045 11697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:45:17.119308 11681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:45:17.119458 11681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "4508a416630345649157e09dd4be2a33"
format_stamp: "Formatted at 2026-09-26 20:45:17 on dist-test-slave-4hd3"
server_key: "0376af3a4edc37139b122fde7bcccf5b"
server_key_iv: "5418eb3f779f841c68fbbdcafc024fac"
server_key_version: "kuduclusterkey@0"
I20260926 20:45:17.119624 11681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:45:17.155037 11681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:45:17.157282 11681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:45:17.157454 11681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:45:17.157721 11681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:45:17.158138 11681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:45:17.158193 11681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:45:17.158264 11681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:45:17.158308 11681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:45:17.166728 11681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:46709
I20260926 20:45:17.167299 11681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:45:17.167758 11811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:46709 every 8 connection(s)
I20260926 20:45:17.168207 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 11681
I20260926 20:45:17.168306 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:45:17.168531 393 external_mini_cluster.cc:1428] Setting key 295c851064f61d39b13805f451e6e571
Sep 26 20:45:17 dist-test-slave-4hd3 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260926 20:45:17.179692 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.168233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:40639 (local address 127.0.98.126:38257)
0926 20:45:17.168901 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0926 20:45:17.168903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:45:17.168914 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:45:17.169264 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:45:17.169265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:45:17.169280 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:45:17.169347 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:45:17.170123 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.170928 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.172530 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.172860 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.172941 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:45:17.175627 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:45:17.175629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:45:17.175630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:45:17.175638 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:45:17.178033 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:45:17.178539 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:45:17.178540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:45:17.178541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:45:17.178595 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:45:17.178919 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:45:17.178920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:45:17.178921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:45:17.179062 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:45:17.179127 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:45:17.179498 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 20:45:17.179589 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":99,"threads_started":1}
I20260926 20:45:17.179913 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.168152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38257 (local address 127.0.98.65:40639)
0926 20:45:17.168441 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 20:45:17.168484 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:45:17.169084 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:45:17.169482 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:45:17.169487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:45:17.169729 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:45:17.170014 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:45:17.170020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.171043 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.171044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:45:17.172440 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:45:17.172443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.172951 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.172952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:45:17.173127 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:45:17.173763 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:45:17.173774 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:45:17.175510 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:45:17.178134 (+ 2624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:45:17.178141 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:45:17.178144 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:45:17.178446 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:45:17.178670 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:45:17.178671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:45:17.178672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:45:17.178842 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:45:17.179156 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:45:17.179163 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:45:17.179334 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 20:45:17.179453 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 20:45:17.179614 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260926 20:45:17.180612 11812 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38257
I20260926 20:45:17.180706 11812 heartbeater.cc:464] Registering TS with master...
I20260926 20:45:17.180904 11812 heartbeater.cc:511] Master 127.0.98.126:38257 requested a full tablet report, sending...
I20260926 20:45:17.181414 11600 ts_manager.cc:194] Registered new tserver with Master: 4508a416630345649157e09dd4be2a33 (127.0.98.65:46709)
I20260926 20:45:17.182281 11600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:40639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260926 20:45:17.198985 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:54443/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38257
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:45:17.301792 11818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:45:17.302019 11818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:45:17.302055 11818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 26 20:45:17 dist-test-slave-4hd3 krb5kdc[11124](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790455517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 20:45:17.303449 11818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:45:17.303512 11818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:45:17.303555 11818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:45:17.305020 11818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.84:54443/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:45:17.305262 11818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:45:17.305513 11818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:45:17.311079 11826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:45:17.311254 11828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:45:17.311072 11825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:45:17.312232 11818 server_base.cc:1034] running on GCE node
I20260926 20:45:17.312439 11818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:45:17.312703 11818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:45:17.313879 11818 hybrid_clock.cc:648] HybridClock initialized: now 1790455517313858 us; error 34 us; skew 500 ppm
Sep 26 20:45:17 dist-test-slave-4hd3 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:45:17.317101 11818 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:45:17.317703 11818 webserver.cc:460] Webserver started at http://127.0.98.66:40981/ using document root <none> and password file <none>
I20260926 20:45:17.317988 11818 fs_manager.cc:359] Metadata directory not provided
I20260926 20:45:17.318099 11818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:45:17.318279 11818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:45:17 dist-test-slave-4hd3 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260926 20:45:17.337123 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "959e2d7ee0574494b8681af0f02cd0c3"
format_stamp: "Formatted at 2026-09-26 20:45:17 on dist-test-slave-4hd3"
server_key: "e6d46d53cc31e52b365c1405835e8795"
server_key_iv: "29c1a4fe71d1c9174439b6bb68a22e6c"
server_key_version: "kuduclusterkey@0"
I20260926 20:45:17.337714 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "959e2d7ee0574494b8681af0f02cd0c3"
format_stamp: "Formatted at 2026-09-26 20:45:17 on dist-test-slave-4hd3"
server_key: "e6d46d53cc31e52b365c1405835e8795"
server_key_iv: "29c1a4fe71d1c9174439b6bb68a22e6c"
server_key_version: "kuduclusterkey@0"
I20260926 20:45:17.348011 11818 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.001s sys 0.000s
I20260926 20:45:17.360919 11835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:45:17.361171 11818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:45:17.361270 11818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "959e2d7ee0574494b8681af0f02cd0c3"
format_stamp: "Formatted at 2026-09-26 20:45:17 on dist-test-slave-4hd3"
server_key: "e6d46d53cc31e52b365c1405835e8795"
server_key_iv: "29c1a4fe71d1c9174439b6bb68a22e6c"
server_key_version: "kuduclusterkey@0"
I20260926 20:45:17.361398 11818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:45:17.381126 11818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:45:17.383662 11818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:45:17.383899 11818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:45:17.384196 11818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:45:17.384595 11818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:45:17.384681 11818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:45:17.384753 11818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:45:17.384812 11818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:45:17.391353 11818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:45783
I20260926 20:45:17.391893 11818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:45:17.394455 11949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:45783 every 8 connection(s)
I20260926 20:45:17.398737 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 11818
I20260926 20:45:17.398862 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:45:17.399055 393 external_mini_cluster.cc:1428] Setting key ccfe4779e61bcf011c763e2fa974adbf
Sep 26 20:45:17 dist-test-slave-4hd3 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:45:17.406406 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.395264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:59245 (local address 127.0.98.126:38257)
0926 20:45:17.395319 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 20:45:17.395328 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:45:17.395830 (+ 502us) server_negotiation.cc:407] Connection header received
0926 20:45:17.396551 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:45:17.396553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:45:17.396565 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:45:17.396598 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:45:17.397445 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.398208 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.400257 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.400563 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.400608 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:45:17.403118 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:45:17.403120 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:45:17.403120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:45:17.403127 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:45:17.404830 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:45:17.405299 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:45:17.405300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:45:17.405300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:45:17.405367 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:45:17.405672 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:45:17.405672 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:45:17.405673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:45:17.405805 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:45:17.405847 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 20:45:17.406211 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 20:45:17.406295 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:45:17.407112 11952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.394991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38257 (local address 127.0.98.66:59245)
0926 20:45:17.395572 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0926 20:45:17.395740 (+ 168us) client_negotiation.cc:174] Beginning negotiation
0926 20:45:17.396419 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:45:17.396801 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:45:17.396805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:45:17.397049 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:45:17.397332 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:45:17.397338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.398338 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.398339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:45:17.400156 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:45:17.400159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.400697 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.400698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:45:17.400834 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:45:17.401411 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:45:17.401421 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:45:17.403006 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:45:17.404921 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:45:17.404927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:45:17.404929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:45:17.405212 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:45:17.405467 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:45:17.405468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:45:17.405468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:45:17.405598 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:45:17.405892 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:45:17.405898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:45:17.406072 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 20:45:17.406726 (+ 654us) client_negotiation.cc:240] Negotiation successful
0926 20:45:17.406876 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":73,"threads_started":1}
I20260926 20:45:17.407778 11950 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38257
I20260926 20:45:17.407864 11950 heartbeater.cc:464] Registering TS with master...
I20260926 20:45:17.408034 11950 heartbeater.cc:511] Master 127.0.98.126:38257 requested a full tablet report, sending...
I20260926 20:45:17.408439 11600 ts_manager.cc:194] Registered new tserver with Master: 959e2d7ee0574494b8681af0f02cd0c3 (127.0.98.66:45783)
I20260926 20:45:17.408923 11600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:59245
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260926 20:45:17.438647 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:54443/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38257
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:45:17.561530 11955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:45:17.561832 11955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:45:17.561878 11955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:45:17.563791 11955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:45:17.563895 11955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:45:17.563964 11955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:45:17.566366 11955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.84:54443/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:45:17.566640 11955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:45:17.566982 11955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:45:17.573797 11963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:45:17.574960 11955 server_base.cc:1034] running on GCE node
W20260926 20:45:17.575071 11961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:45:17.575259 11960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:45:17.575625 11955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:45:17.575943 11955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:45:17.577131 11955 hybrid_clock.cc:648] HybridClock initialized: now 1790455517577090 us; error 47 us; skew 500 ppm
Sep 26 20:45:17 dist-test-slave-4hd3 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:45:17.580003 11955 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:45:17.580524 11955 webserver.cc:460] Webserver started at http://127.0.98.67:41687/ using document root <none> and password file <none>
I20260926 20:45:17.580758 11955 fs_manager.cc:359] Metadata directory not provided
I20260926 20:45:17.580824 11955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:45:17.580981 11955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:45:17 dist-test-slave-4hd3 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260926 20:45:17.593499 11955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "c590e594ac564c06834d9c1cc4ecca24"
format_stamp: "Formatted at 2026-09-26 20:45:17 on dist-test-slave-4hd3"
server_key: "235a628449f276c185e440ac8265af48"
server_key_iv: "920acf34c463a6f9a3c492ccf8b9abf7"
server_key_version: "kuduclusterkey@0"
I20260926 20:45:17.594046 11955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "c590e594ac564c06834d9c1cc4ecca24"
format_stamp: "Formatted at 2026-09-26 20:45:17 on dist-test-slave-4hd3"
server_key: "235a628449f276c185e440ac8265af48"
server_key_iv: "920acf34c463a6f9a3c492ccf8b9abf7"
server_key_version: "kuduclusterkey@0"
I20260926 20:45:17.600122 11955 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.000s
I20260926 20:45:17.624769 11970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:45:17.624974 11955 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.000s sys 0.001s
I20260926 20:45:17.625072 11955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "c590e594ac564c06834d9c1cc4ecca24"
format_stamp: "Formatted at 2026-09-26 20:45:17 on dist-test-slave-4hd3"
server_key: "235a628449f276c185e440ac8265af48"
server_key_iv: "920acf34c463a6f9a3c492ccf8b9abf7"
server_key_version: "kuduclusterkey@0"
I20260926 20:45:17.625178 11955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:45:17.659222 11955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:45:17.661729 11955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:45:17.661899 11955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:45:17.662182 11955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:45:17.662540 11955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:45:17.662621 11955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:45:17.662689 11955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:45:17.662719 11955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:45:17.670312 11955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:42013
I20260926 20:45:17.670811 11955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:45:17.670845 12083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:42013 every 8 connection(s)
I20260926 20:45:17.676473 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 11955
I20260926 20:45:17.676584 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:45:17.676795 393 external_mini_cluster.cc:1428] Setting key 097048ae63d85cebafce6a86a84f8562
Sep 26 20:45:17 dist-test-slave-4hd3 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:45:17.683624 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.672289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:42209 (local address 127.0.98.126:38257)
0926 20:45:17.672336 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:45:17.672339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:45:17.672600 (+ 261us) server_negotiation.cc:407] Connection header received
0926 20:45:17.673123 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:45:17.673125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:45:17.673139 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:45:17.673198 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:45:17.673878 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.674389 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.675400 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.675614 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.675693 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:45:17.680209 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:45:17.680211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:45:17.680211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:45:17.680220 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:45:17.681922 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:45:17.682437 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:45:17.682438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:45:17.682439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:45:17.682503 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:45:17.682826 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:45:17.682826 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:45:17.682827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:45:17.682980 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:45:17.683050 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:45:17.683444 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 20:45:17.683529 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:45:17.684367 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.672294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38257 (local address 127.0.98.67:42209)
0926 20:45:17.672449 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 20:45:17.672491 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:45:17.673003 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:45:17.673271 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:45:17.673275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:45:17.673555 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:45:17.673783 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:45:17.673788 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.674492 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.674493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:45:17.675297 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:45:17.675299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.675699 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.675700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:45:17.675853 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:45:17.677026 (+ 1173us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:45:17.677038 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:45:17.680043 (+ 3005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:45:17.682019 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:45:17.682025 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:45:17.682028 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:45:17.682341 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:45:17.682585 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:45:17.682586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:45:17.682586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:45:17.682741 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:45:17.683097 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:45:17.683104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:45:17.683278 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 20:45:17.683975 (+ 697us) client_negotiation.cc:240] Negotiation successful
0926 20:45:17.684139 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260926 20:45:17.685012 12084 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38257
I20260926 20:45:17.685118 12084 heartbeater.cc:464] Registering TS with master...
I20260926 20:45:17.685309 12084 heartbeater.cc:511] Master 127.0.98.126:38257 requested a full tablet report, sending...
I20260926 20:45:17.685760 11600 ts_manager.cc:194] Registered new tserver with Master: c590e594ac564c06834d9c1cc4ecca24 (127.0.98.67:42013)
I20260926 20:45:17.686271 11600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:42209
I20260926 20:45:17.688553 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:45:17.700021 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.692005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53614 (local address 127.0.98.126:38257)
0926 20:45:17.692044 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:45:17.692046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:45:17.692194 (+ 148us) server_negotiation.cc:407] Connection header received
0926 20:45:17.692245 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:45:17.692247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:45:17.692258 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:45:17.692320 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:45:17.693000 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.693767 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.694852 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.695131 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.695205 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:45:17.696801 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:45:17.696803 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:45:17.696803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:45:17.696811 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:45:17.698653 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:45:17.699073 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:45:17.699074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:45:17.699074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:45:17.699131 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:45:17.699351 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:45:17.699352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:45:17.699352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:45:17.699475 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:45:17.699549 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:45:17.699837 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 20:45:17.699928 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:45:17.702026 11600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:45:17.704340 11600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:45:17.718777 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.708916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:42013 (local address 127.0.0.1:58100)
0926 20:45:17.709359 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 20:45:17.709404 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:45:17.709474 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:45:17.710102 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:45:17.710105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:45:17.710108 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:45:17.710186 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:45:17.710188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.715820 (+ 5632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.715822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:45:17.717196 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:45:17.717199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.718455 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.718456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:45:17.718555 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:45:17.718573 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 20:45:17.718623 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 20:45:17.718676 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":76,"threads_started":1}
I20260926 20:45:17.719309 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.709454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58100 (local address 127.0.98.67:42013)
0926 20:45:17.709765 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 20:45:17.709770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:45:17.709917 (+ 147us) server_negotiation.cc:407] Connection header received
0926 20:45:17.709965 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:45:17.709968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:45:17.709984 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:45:17.710267 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:45:17.710335 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.715650 (+ 5315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.717325 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.718350 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.719079 (+ 729us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:45:17.719168 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:45:17.719194 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 20:45:17.719231 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":74,"threads_started":1}
I20260926 20:45:17.720119 12018 tablet_service.cc:1467] Processing CreateTablet for tablet 30fb9bd65fec46d5aac756d6562ebaa8 (DEFAULT_TABLE table=test-table [id=48a81c9fc70848c98561c161d39758b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:45:17.720515 12018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30fb9bd65fec46d5aac756d6562ebaa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:45:17.720640 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.709154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:45783 (local address 127.0.0.1:54632)
0926 20:45:17.709588 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 20:45:17.709597 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:45:17.709634 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:45:17.715788 (+ 6154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:45:17.715789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:45:17.715792 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:45:17.715852 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:45:17.715853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.717614 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.717615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:45:17.718809 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:45:17.718812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.720407 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.720408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:45:17.720485 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:45:17.720489 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:45:17.720522 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 20:45:17.720567 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":1,"thread_start_us":66,"threads_started":1}
I20260926 20:45:17.721251 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.709610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54632 (local address 127.0.98.66:45783)
0926 20:45:17.715569 (+ 5959us) server_negotiation.cc:206] Beginning negotiation
0926 20:45:17.715574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:45:17.715586 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:45:17.715627 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:45:17.715629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:45:17.715646 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:45:17.715925 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:45:17.716000 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.717510 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.719213 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.720268 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.720980 (+ 712us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:45:17.721066 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 20:45:17.721094 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 20:45:17.721132 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5918,"thread_start_us":81,"threads_started":1}
I20260926 20:45:17.721994 11884 tablet_service.cc:1467] Processing CreateTablet for tablet 30fb9bd65fec46d5aac756d6562ebaa8 (DEFAULT_TABLE table=test-table [id=48a81c9fc70848c98561c161d39758b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:45:17.722350 11884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30fb9bd65fec46d5aac756d6562ebaa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:45:17.724428 12103 tablet_bootstrap.cc:492] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24: Bootstrap starting.
I20260926 20:45:17.725442 12103 tablet_bootstrap.cc:654] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24: Neither blocks nor log segments found. Creating new log.
I20260926 20:45:17.725747 12103 log.cc:824] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24: Log is configured to *not* fsync() on all Append() calls
I20260926 20:45:17.726505 12103 tablet_bootstrap.cc:492] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24: No bootstrap required, opened a new log
I20260926 20:45:17.726577 12103 ts_tablet_manager.cc:1397] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:45:17.728485 12103 raft_consensus.cc:348] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c590e594ac564c06834d9c1cc4ecca24" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42013 } } peers { permanent_uuid: "959e2d7ee0574494b8681af0f02cd0c3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45783 } } peers { permanent_uuid: "4508a416630345649157e09dd4be2a33" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46709 } }
I20260926 20:45:17.728657 12103 raft_consensus.cc:374] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:45:17.728699 12103 raft_consensus.cc:729] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c590e594ac564c06834d9c1cc4ecca24, State: Initialized, Role: FOLLOWER
I20260926 20:45:17.728820 12103 consensus_queue.cc:260] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c590e594ac564c06834d9c1cc4ecca24" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42013 } } peers { permanent_uuid: "959e2d7ee0574494b8681af0f02cd0c3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45783 } } peers { permanent_uuid: "4508a416630345649157e09dd4be2a33" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46709 } }
I20260926 20:45:17.728989 12105 tablet_bootstrap.cc:492] T 30fb9bd65fec46d5aac756d6562ebaa8 P 959e2d7ee0574494b8681af0f02cd0c3: Bootstrap starting.
I20260926 20:45:17.729090 12103 ts_tablet_manager.cc:1428] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:45:17.729501 12084 heartbeater.cc:503] Master 127.0.98.126:38257 was elected leader, sending a full tablet report...
I20260926 20:45:17.732913 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.709840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:46709 (local address 127.0.0.1:46024)
0926 20:45:17.718955 (+ 9115us) negotiation.cc:107] Waiting for socket to connect
0926 20:45:17.718965 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:45:17.718998 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:45:17.727316 (+ 8318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:45:17.727317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:45:17.727319 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:45:17.727356 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:45:17.727358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.729725 (+ 2367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.729726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:45:17.730896 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:45:17.730900 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.732649 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.732650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:45:17.732717 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:45:17.732721 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:45:17.732794 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 20:45:17.732836 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9104,"thread_start_us":76,"threads_started":1}
I20260926 20:45:17.733141 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.710044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46024 (local address 127.0.98.65:46709)
0926 20:45:17.727143 (+ 17099us) server_negotiation.cc:206] Beginning negotiation
0926 20:45:17.727148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:45:17.727160 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:45:17.727194 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:45:17.727196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:45:17.727213 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:45:17.727442 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:45:17.727511 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.729625 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.731750 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.732552 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.732632 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:45:17.732687 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:45:17.733026 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 20:45:17.733073 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17063,"thread_start_us":56,"threads_started":1}
I20260926 20:45:17.733819 12105 tablet_bootstrap.cc:654] T 30fb9bd65fec46d5aac756d6562ebaa8 P 959e2d7ee0574494b8681af0f02cd0c3: Neither blocks nor log segments found. Creating new log.
I20260926 20:45:17.733847 11746 tablet_service.cc:1467] Processing CreateTablet for tablet 30fb9bd65fec46d5aac756d6562ebaa8 (DEFAULT_TABLE table=test-table [id=48a81c9fc70848c98561c161d39758b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:45:17.734177 11746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30fb9bd65fec46d5aac756d6562ebaa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:45:17.734338 12105 log.cc:824] T 30fb9bd65fec46d5aac756d6562ebaa8 P 959e2d7ee0574494b8681af0f02cd0c3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:45:17.735097 12105 tablet_bootstrap.cc:492] T 30fb9bd65fec46d5aac756d6562ebaa8 P 959e2d7ee0574494b8681af0f02cd0c3: No bootstrap required, opened a new log
I20260926 20:45:17.735160 12105 ts_tablet_manager.cc:1397] T 30fb9bd65fec46d5aac756d6562ebaa8 P 959e2d7ee0574494b8681af0f02cd0c3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 20:45:17.737211 12105 raft_consensus.cc:348] T 30fb9bd65fec46d5aac756d6562ebaa8 P 959e2d7ee0574494b8681af0f02cd0c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c590e594ac564c06834d9c1cc4ecca24" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42013 } } peers { permanent_uuid: "959e2d7ee0574494b8681af0f02cd0c3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45783 } } peers { permanent_uuid: "4508a416630345649157e09dd4be2a33" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46709 } }
I20260926 20:45:17.737376 12105 raft_consensus.cc:374] T 30fb9bd65fec46d5aac756d6562ebaa8 P 959e2d7ee0574494b8681af0f02cd0c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:45:17.737416 12105 raft_consensus.cc:729] T 30fb9bd65fec46d5aac756d6562ebaa8 P 959e2d7ee0574494b8681af0f02cd0c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 959e2d7ee0574494b8681af0f02cd0c3, State: Initialized, Role: FOLLOWER
I20260926 20:45:17.737525 12105 consensus_queue.cc:260] T 30fb9bd65fec46d5aac756d6562ebaa8 P 959e2d7ee0574494b8681af0f02cd0c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c590e594ac564c06834d9c1cc4ecca24" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42013 } } peers { permanent_uuid: "959e2d7ee0574494b8681af0f02cd0c3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45783 } } peers { permanent_uuid: "4508a416630345649157e09dd4be2a33" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46709 } }
I20260926 20:45:17.737756 12105 ts_tablet_manager.cc:1428] T 30fb9bd65fec46d5aac756d6562ebaa8 P 959e2d7ee0574494b8681af0f02cd0c3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:45:17.738050 11950 heartbeater.cc:503] Master 127.0.98.126:38257 was elected leader, sending a full tablet report...
I20260926 20:45:17.738204 12108 tablet_bootstrap.cc:492] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33: Bootstrap starting.
I20260926 20:45:17.738870 12108 tablet_bootstrap.cc:654] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33: Neither blocks nor log segments found. Creating new log.
I20260926 20:45:17.739616 12108 log.cc:824] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33: Log is configured to *not* fsync() on all Append() calls
I20260926 20:45:17.740485 12108 tablet_bootstrap.cc:492] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33: No bootstrap required, opened a new log
I20260926 20:45:17.740559 12108 ts_tablet_manager.cc:1397] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:45:17.743400 12108 raft_consensus.cc:348] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c590e594ac564c06834d9c1cc4ecca24" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42013 } } peers { permanent_uuid: "959e2d7ee0574494b8681af0f02cd0c3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45783 } } peers { permanent_uuid: "4508a416630345649157e09dd4be2a33" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46709 } }
I20260926 20:45:17.743573 12108 raft_consensus.cc:374] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:45:17.743620 12108 raft_consensus.cc:729] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4508a416630345649157e09dd4be2a33, State: Initialized, Role: FOLLOWER
I20260926 20:45:17.743780 12108 consensus_queue.cc:260] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c590e594ac564c06834d9c1cc4ecca24" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42013 } } peers { permanent_uuid: "959e2d7ee0574494b8681af0f02cd0c3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45783 } } peers { permanent_uuid: "4508a416630345649157e09dd4be2a33" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46709 } }
I20260926 20:45:17.744040 12108 ts_tablet_manager.cc:1428] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260926 20:45:17.744079 11812 heartbeater.cc:503] Master 127.0.98.126:38257 was elected leader, sending a full tablet report...
Sep 26 20:45:17 dist-test-slave-4hd3 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
W20260926 20:45:17.895820 11951 tablet.cc:2369] T 30fb9bd65fec46d5aac756d6562ebaa8 P 959e2d7ee0574494b8681af0f02cd0c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:45:17.899333 12111 raft_consensus.cc:482] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:45:17.899473 12111 raft_consensus.cc:504] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c590e594ac564c06834d9c1cc4ecca24" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42013 } } peers { permanent_uuid: "959e2d7ee0574494b8681af0f02cd0c3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45783 } } peers { permanent_uuid: "4508a416630345649157e09dd4be2a33" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46709 } }
I20260926 20:45:17.899904 12111 leader_election.cc:288] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c590e594ac564c06834d9c1cc4ecca24 (127.0.98.67:42013), 959e2d7ee0574494b8681af0f02cd0c3 (127.0.98.66:45783)
I20260926 20:45:17.904055 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.900276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:42415 (local address 127.0.98.66:45783)
0926 20:45:17.900329 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:45:17.900331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:45:17.900474 (+ 143us) server_negotiation.cc:407] Connection header received
0926 20:45:17.900581 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:45:17.900582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:45:17.900594 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:45:17.900624 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:45:17.900835 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.901566 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.903108 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.903649 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.903701 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:45:17.903709 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:45:17.903952 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 20:45:17.903988 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 20:45:17.904055 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.900242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:48247 (local address 127.0.98.67:42013)
0926 20:45:17.900291 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 20:45:17.900293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:45:17.900357 (+ 64us) server_negotiation.cc:407] Connection header received
0926 20:45:17.900494 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:45:17.900495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:45:17.900505 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:45:17.900546 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:45:17.900806 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.901566 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.903101 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.903649 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.903695 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:45:17.903705 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:45:17.903952 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 20:45:17.903988 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:45:17.904317 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.900045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:42013 (local address 127.0.98.65:48247)
0926 20:45:17.900293 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 20:45:17.900304 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:45:17.900407 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:45:17.900622 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:45:17.900623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:45:17.900624 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:45:17.900719 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:45:17.900722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.901688 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.901688 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:45:17.902994 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:45:17.902997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.903789 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.903789 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:45:17.903865 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:45:17.903870 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:45:17.904169 (+ 299us) client_negotiation.cc:240] Negotiation successful
0926 20:45:17.904238 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":42,"thread_start_us":90,"threads_started":1}
I20260926 20:45:17.904620 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.900062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:45783 (local address 127.0.98.65:42415)
0926 20:45:17.900418 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 20:45:17.900426 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:45:17.900495 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:45:17.900696 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:45:17.900696 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:45:17.900698 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:45:17.900756 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:45:17.900757 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.901696 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.901697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:45:17.902995 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:45:17.902997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.903787 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.903788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:45:17.903865 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:45:17.903868 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:45:17.904517 (+ 649us) client_negotiation.cc:240] Negotiation successful
0926 20:45:17.904552 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":59,"thread_start_us":90,"threads_started":1}
I20260926 20:45:17.904616 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fb9bd65fec46d5aac756d6562ebaa8" candidate_uuid: "4508a416630345649157e09dd4be2a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c590e594ac564c06834d9c1cc4ecca24" is_pre_election: true
I20260926 20:45:17.904783 12038 raft_consensus.cc:2454] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4508a416630345649157e09dd4be2a33 in term 0.
I20260926 20:45:17.904945 11901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fb9bd65fec46d5aac756d6562ebaa8" candidate_uuid: "4508a416630345649157e09dd4be2a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "959e2d7ee0574494b8681af0f02cd0c3" is_pre_election: true
I20260926 20:45:17.905045 11702 leader_election.cc:302] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4508a416630345649157e09dd4be2a33, c590e594ac564c06834d9c1cc4ecca24; no voters:
I20260926 20:45:17.905078 11901 raft_consensus.cc:2454] T 30fb9bd65fec46d5aac756d6562ebaa8 P 959e2d7ee0574494b8681af0f02cd0c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4508a416630345649157e09dd4be2a33 in term 0.
I20260926 20:45:17.905155 12111 raft_consensus.cc:2781] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:45:17.905200 12111 raft_consensus.cc:482] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:45:17.905227 12111 raft_consensus.cc:3037] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:45:17.905951 12111 raft_consensus.cc:504] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c590e594ac564c06834d9c1cc4ecca24" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42013 } } peers { permanent_uuid: "959e2d7ee0574494b8681af0f02cd0c3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45783 } } peers { permanent_uuid: "4508a416630345649157e09dd4be2a33" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46709 } }
I20260926 20:45:17.906091 12111 leader_election.cc:288] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 [CANDIDATE]: Term 1 election: Requested vote from peers c590e594ac564c06834d9c1cc4ecca24 (127.0.98.67:42013), 959e2d7ee0574494b8681af0f02cd0c3 (127.0.98.66:45783)
I20260926 20:45:17.906263 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fb9bd65fec46d5aac756d6562ebaa8" candidate_uuid: "4508a416630345649157e09dd4be2a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c590e594ac564c06834d9c1cc4ecca24"
I20260926 20:45:17.906366 12038 raft_consensus.cc:3037] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:45:17.906337 11901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fb9bd65fec46d5aac756d6562ebaa8" candidate_uuid: "4508a416630345649157e09dd4be2a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "959e2d7ee0574494b8681af0f02cd0c3"
I20260926 20:45:17.906437 11901 raft_consensus.cc:3037] T 30fb9bd65fec46d5aac756d6562ebaa8 P 959e2d7ee0574494b8681af0f02cd0c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:45:17.907044 12038 raft_consensus.cc:2454] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4508a416630345649157e09dd4be2a33 in term 1.
I20260926 20:45:17.907214 11702 leader_election.cc:302] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4508a416630345649157e09dd4be2a33, c590e594ac564c06834d9c1cc4ecca24; no voters:
I20260926 20:45:17.907279 11901 raft_consensus.cc:2454] T 30fb9bd65fec46d5aac756d6562ebaa8 P 959e2d7ee0574494b8681af0f02cd0c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4508a416630345649157e09dd4be2a33 in term 1.
I20260926 20:45:17.907317 12111 raft_consensus.cc:2781] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:45:17.907495 12111 raft_consensus.cc:686] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 [term 1 LEADER]: Becoming Leader. State: Replica: 4508a416630345649157e09dd4be2a33, State: Running, Role: LEADER
I20260926 20:45:17.907584 12111 consensus_queue.cc:237] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c590e594ac564c06834d9c1cc4ecca24" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42013 } } peers { permanent_uuid: "959e2d7ee0574494b8681af0f02cd0c3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45783 } } peers { permanent_uuid: "4508a416630345649157e09dd4be2a33" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46709 } }
I20260926 20:45:17.908386 11600 catalog_manager.cc:5495] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4508a416630345649157e09dd4be2a33 (127.0.98.65). New cstate: current_term: 1 leader_uuid: "4508a416630345649157e09dd4be2a33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c590e594ac564c06834d9c1cc4ecca24" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "959e2d7ee0574494b8681af0f02cd0c3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4508a416630345649157e09dd4be2a33" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46709 } health_report { overall_health: HEALTHY } } }
W20260926 20:45:17.918326 11813 tablet.cc:2369] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:45:17.922250 12085 tablet.cc:2369] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:45:17.947613 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.945135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46030 (local address 127.0.98.65:46709)
0926 20:45:17.945204 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 20:45:17.945206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:45:17.945214 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:45:17.945258 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:45:17.945259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:45:17.945279 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:45:17.945360 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:45:17.945502 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.945986 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.946767 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:45:17.946982 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:45:17.947061 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:45:17.947234 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:45:17.947354 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:45:17.947457 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:45:17.947475 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 20:45:17.947497 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:45:17.949653 12038 raft_consensus.cc:1261] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24 [term 1 FOLLOWER]: Refusing update from remote peer 4508a416630345649157e09dd4be2a33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:45:17.949673 11901 raft_consensus.cc:1261] T 30fb9bd65fec46d5aac756d6562ebaa8 P 959e2d7ee0574494b8681af0f02cd0c3 [term 1 FOLLOWER]: Refusing update from remote peer 4508a416630345649157e09dd4be2a33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:45:17.950037 12114 consensus_queue.cc:1035] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c590e594ac564c06834d9c1cc4ecca24" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:45:17.950183 12111 consensus_queue.cc:1035] T 30fb9bd65fec46d5aac756d6562ebaa8 P 4508a416630345649157e09dd4be2a33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "959e2d7ee0574494b8681af0f02cd0c3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:45:17.952564 12117 mvcc.cc:205] Tried to move back new op lower bound from 7333705801518972928 to 7333705801350172672. Current Snapshot: MvccSnapshot[applied={T|T < 7333705801518972928}]
I20260926 20:45:17.953215 12119 mvcc.cc:205] Tried to move back new op lower bound from 7333705801518972928 to 7333705801350172672. Current Snapshot: MvccSnapshot[applied={T|T < 7333705801518972928}]
I20260926 20:45:17.954686 12118 mvcc.cc:205] Tried to move back new op lower bound from 7333705801518972928 to 7333705801350172672. Current Snapshot: MvccSnapshot[applied={T|T < 7333705801518972928}]
I20260926 20:45:17.956151 11599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:45:17.956264 11599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:45:17.956974 11599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 495255bba4cb437ea28c64c42e6c106c: Sending DeleteTablet for 3 replicas of tablet 30fb9bd65fec46d5aac756d6562ebaa8
I20260926 20:45:17.957418 12018 tablet_service.cc:1514] Processing DeleteTablet for tablet 30fb9bd65fec46d5aac756d6562ebaa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:45:17 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:58100
I20260926 20:45:17.957633 11884 tablet_service.cc:1514] Processing DeleteTablet for tablet 30fb9bd65fec46d5aac756d6562ebaa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:45:17 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:54632
I20260926 20:45:17.958014 11746 tablet_service.cc:1514] Processing DeleteTablet for tablet 30fb9bd65fec46d5aac756d6562ebaa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:45:17 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:46024
I20260926 20:45:17.958598 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 11681
I20260926 20:45:17.961018 12126 tablet_replica.cc:330] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24: stopping tablet replica
I20260926 20:45:17.961146 12126 raft_consensus.cc:2229] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:45:17.961277 12126 raft_consensus.cc:2258] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:45:17.961788 12126 ts_tablet_manager.cc:1905] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:45:17.963534 12126 ts_tablet_manager.cc:1918] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:45:17.963608 12126 log.cc:1196] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/30fb9bd65fec46d5aac756d6562ebaa8
I20260926 20:45:17.964203 12127 tablet_replica.cc:330] T 30fb9bd65fec46d5aac756d6562ebaa8 P 959e2d7ee0574494b8681af0f02cd0c3: stopping tablet replica
I20260926 20:45:17.964234 12126 ts_tablet_manager.cc:1939] T 30fb9bd65fec46d5aac756d6562ebaa8 P c590e594ac564c06834d9c1cc4ecca24: Deleting consensus metadata
I20260926 20:45:17.964327 12127 raft_consensus.cc:2229] T 30fb9bd65fec46d5aac756d6562ebaa8 P 959e2d7ee0574494b8681af0f02cd0c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:45:17.964468 12127 raft_consensus.cc:2258] T 30fb9bd65fec46d5aac756d6562ebaa8 P 959e2d7ee0574494b8681af0f02cd0c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:45:17.964653 11587 catalog_manager.cc:4841] TS c590e594ac564c06834d9c1cc4ecca24 (127.0.98.67:42013): tablet 30fb9bd65fec46d5aac756d6562ebaa8 (table test-table [id=48a81c9fc70848c98561c161d39758b7]) successfully deleted
W20260926 20:45:17.965256 11585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:46709 (error 108)
I20260926 20:45:17.965564 12127 ts_tablet_manager.cc:1905] T 30fb9bd65fec46d5aac756d6562ebaa8 P 959e2d7ee0574494b8681af0f02cd0c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:45:17.965586 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 11818
I20260926 20:45:17.966442 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.965758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:46709 (local address 127.0.0.1:46032)
0926 20:45:17.966380 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0926 20:45:17.966414 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":594}
W20260926 20:45:17.968261 11585 catalog_manager.cc:4584] TS 4508a416630345649157e09dd4be2a33 (127.0.98.65:46709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 30fb9bd65fec46d5aac756d6562ebaa8: Network error: Client connection negotiation failed: client connection to 127.0.98.65:46709: connect: Connection refused (error 111)
I20260926 20:45:17.971624 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:45:17.971524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:45783 (local address 127.0.0.1:54638)
0926 20:45:17.971560 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 20:45:17.971585 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.66:45783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 20:45:17.971957 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 11955
I20260926 20:45:17.976779 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 11568
I20260926 20:45:17.982319 393 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 20:45:18.316630 393 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 20:45:18.316744 393 mini_ranger.cc:66] Stopping Ranger...
I20260926 20:45:18.661054 393 mini_ranger.cc:68] Stopped Ranger
2026-09-26 20:45:18.661 UTC [11154] LOG: received smart shutdown request
2026-09-26 20:45:18.664 UTC [11154] LOG: background worker "logical replication launcher" (PID 11163) exited with exit code 1
2026-09-26 20:45:18.665 UTC [11158] LOG: shutting down
2026-09-26 20:45:18.836 UTC [11154] LOG: database system is shut down
2026-09-26T20:45:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45959 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:45:19 dist-test-slave-4hd3 krb5kdc[12138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:45:19 dist-test-slave-4hd3 krb5kdc[12138](info): set up 2 sockets
Sep 26 20:45:19 dist-test-slave-4hd3 krb5kdc[12138](info): commencing operation
krb5kdc: starting...
W20260926 20:45:19.975589 393 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:45:20 dist-test-slave-4hd3 krb5kdc[12138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:45:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:45:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/rangeradmin_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/rangeradmin_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/rangerlookup_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/rangerlookup_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
I20260926 20:45:20.130271 393 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDRmUyo/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/postgres -l logfile start
2026-09-26 20:45:22.587 UTC [12168] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 20:45:22.588 UTC [12168] LOG: listening on IPv4 address "127.0.98.84", port 40423
2026-09-26 20:45:22.591 UTC [12168] LOG: listening on Unix socket "/tmp/.s.PGSQL.40423"
2026-09-26 20:45:22.612 UTC [12171] LOG: database system was shut down at 2026-09-26 20:45:21 UTC
2026-09-26 20:45:22.616 UTC [12168] LOG: database system is ready to accept connections
I20260926 20:45:23.559701 393 mini_postgres.cc:96] Postgres bound to 40423
2026-09-26 20:45:23.566 UTC [12179] FATAL: database "slave" does not exist
127.0.98.84:40423 - accepting connections
I20260926 20:45:23.567087 393 mini_ranger.cc:161] Starting Ranger...
I20260926 20:45:23.581923 393 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 20:45:24.050824 393 mini_ranger.cc:87] Created ranger Postgres database
I20260926 20:45:24.050892 393 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ranger-admin
2026-09-26 20:45:24,322 [I] DB FLAVOR :POSTGRES
2026-09-26 20:45:24,323 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:45:24,323 [I] Checking connection..
2026-09-26 20:45:24,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 20:45:24,567 [I] Checking connection passed.
2026-09-26 20:45:24,568 [I] --------- Verifying version history table ---------
2026-09-26 20:45:24,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:45:24,795 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:45:24,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:45:25,049 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:45:25,050 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 20:45:25,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 20:45:25.285 UTC [12243] WARNING: there is no transaction in progress
2026-09-26 20:45:25,291 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 20:45:25,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:45:25,524 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 20:45:25,524 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:45:25,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 20:45:25,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 20:45:26,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4hd3.c.gcp-upstream.internal','N') ;"
2026-09-26 20:45:26,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:45:26,546 [I] Table x_portal_user does not exist in database ranger
2026-09-26 20:45:26,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:45:26,790 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 20:45:26,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:45:27,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:45:27,255 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 20:45:27,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 20:45:28.209 UTC [12351] WARNING: there is no transaction in progress
2026-09-26 20:45:28.225 UTC [12351] WARNING: there is no transaction in progress
2026-09-26 20:45:28.241 UTC [12351] WARNING: there is no transaction in progress
2026-09-26 20:45:28.258 UTC [12351] WARNING: there is no transaction in progress
2026-09-26 20:45:28.276 UTC [12351] WARNING: there is no transaction in progress
2026-09-26 20:45:28.293 UTC [12351] WARNING: there is no transaction in progress
2026-09-26 20:45:28.348 UTC [12351] WARNING: there is no transaction in progress
2026-09-26 20:45:28.355 UTC [12351] WARNING: there is no transaction in progress
2026-09-26 20:45:28.370 UTC [12351] WARNING: there is no transaction in progress
2026-09-26 20:45:28.381 UTC [12351] WARNING: there is no transaction in progress
2026-09-26 20:45:28.391 UTC [12351] WARNING: there is no transaction in progress
2026-09-26 20:45:28.404 UTC [12351] WARNING: there is no transaction in progress
2026-09-26 20:45:28.414 UTC [12351] WARNING: there is no transaction in progress
2026-09-26 20:45:28.420 UTC [12351] WARNING: there is no transaction in progress
2026-09-26 20:45:28.427 UTC [12351] WARNING: there is no transaction in progress
2026-09-26 20:45:28.434 UTC [12351] WARNING: there is no transaction in progress
2026-09-26 20:45:29,076 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 20:45:29,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 20:45:29,329 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 20:45:29,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:45:29,591 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 20:45:29,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:45:29,852 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 20:45:29,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:45:30,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:45:30,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4hd3.c.gcp-upstream.internal';"
2026-09-26 20:45:30,582 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 20:45:30,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 20:45:30,830 [I] DB_PATCHES have already been applied
I20260926 20:45:30.835685 393 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDRmUyo/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 20:45:30.835791 393 mini_ranger.cc:193] Using host: 127.0.98.84
I20260926 20:45:30.838815 393 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.84:60885
Sep 26, 2026 8:45:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:45:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:45:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 8:45:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 8:45:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 8:45:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 8:45:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 20:45:31.945238 393 mini_ranger.cc:160] Time spent starting Ranger: real 8.378s user 0.000s sys 0.008s
Sep 26, 2026 8:45:52 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 8:45:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 8:45:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 8:45:52 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 8:45:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 8:45:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 8:45:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 8:45:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 8:45:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 8:45:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 8:45:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 8:45:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 8:45:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 8:45:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 8:45:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 8:45:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 8:45:52 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 8:45:53 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 20:45:53.630050 393 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/rangerkms_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/rangerkms_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 20:45:53 dist-test-slave-4hd3 krb5kdc[12138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455553, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 20:45:53.722702 393 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 20:45:53.731491 393 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 20:45:54.525543 393 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 20:45:54.525625 393 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ranger-kms
I20260926 20:45:54.525643 393 mini_ranger_kms.cc:227] Using postgres at 127.0.98.84:40423
2026-09-26 20:45:54,687 [I] DB FLAVOR :POSTGRES
2026-09-26 20:45:54,688 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:45:54,688 [I] Checking connection
2026-09-26 20:45:54,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 20:45:54,922 [I] connection success
2026-09-26 20:45:54,922 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 20:45:54,922 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 20:45:54,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 20:45:55,167 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 20:45:55,167 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:45:55,167 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 20:45:55,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:40423/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 20:45:55,429 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 20:45:55.930413 393 mini_ranger_kms.cc:310] Created kms service
I20260926 20:45:56.107384 393 mini_ranger_kms.cc:326] Created kudu user
I20260926 20:45:56.153710 393 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 20:45:56 dist-test-slave-4hd3 krb5kdc[12138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455553, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260926 20:45:56.562357 393 mini_ranger_kms.cc:384] Added ranger policy
I20260926 20:45:56.562577 393 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDRmUyo/build/release/bin/postgresql.jar:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDRmUyo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 20:45:56.562604 393 mini_ranger_kms.cc:240] Using host: 127.0.98.84
I20260926 20:45:56.565979 393 mini_ranger_kms.cc:276] Ranger KMS PID: 12531
I20260926 20:45:56.566069 393 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.84:37999
Sep 26, 2026 8:45:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:45:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:45:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 8:45:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 8:45:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 8:45:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 20:45:58 dist-test-slave-4hd3 krb5kdc[12138](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790455558, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:45:58.819998 393 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.097s user 0.000s sys 0.143s
I20260926 20:45:58.820116 393 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 20:45:58.820171 393 mini_ranger_kms.cc:401] 127.0.98.84:37999/kms/v1/keys
Sep 26 20:45:59 dist-test-slave-4hd3 krb5kdc[12138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455558, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.84@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
Sep 26 20:45:59 dist-test-slave-4hd3 krb5kdc[12138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455558, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.84@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
Sep 26 20:46:01 dist-test-slave-4hd3 krb5kdc[12138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:46:01.710227 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40093
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:40093
--ranger_config_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:37999/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:46:01.791038 12582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:01.791235 12582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:01.791263 12582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:01.792443 12582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:46:01.792493 12582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:01.792507 12582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:01.792519 12582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:46:01.792531 12582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:46:01.794087 12582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.84:37999/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:40093
--ranger_config_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:01.794320 12582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:01.794550 12582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:01.799494 12587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:01.799597 12582 server_base.cc:1034] running on GCE node
W20260926 20:46:01.799760 12588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:01.799494 12590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:01.800062 12582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:01.800386 12582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:01.801544 12582 hybrid_clock.cc:648] HybridClock initialized: now 1790455561801533 us; error 27 us; skew 500 ppm
Sep 26 20:46:01 dist-test-slave-4hd3 krb5kdc[12138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:01.808758 12582 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:46:01.809278 12582 webserver.cc:460] Webserver started at http://127.0.98.126:41239/ using document root <none> and password file <none>
I20260926 20:46:01.809494 12582 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:01.809533 12582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:01.809659 12582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:46:01 dist-test-slave-4hd3 krb5kdc[12138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260926 20:46:01.869096 12582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "57879009e1c6483fb5ed27868f10bd77"
format_stamp: "Formatted at 2026-09-26 20:46:01 on dist-test-slave-4hd3"
server_key: "af6cb938897ced445c4c3a758f5289a6"
server_key_iv: "d62c661587d2b3a773dda83220189946"
server_key_version: "kuduclusterkey@0"
I20260926 20:46:01.869509 12582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "57879009e1c6483fb5ed27868f10bd77"
format_stamp: "Formatted at 2026-09-26 20:46:01 on dist-test-slave-4hd3"
server_key: "af6cb938897ced445c4c3a758f5289a6"
server_key_iv: "d62c661587d2b3a773dda83220189946"
server_key_version: "kuduclusterkey@0"
I20260926 20:46:01.870941 12582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:46:01.884464 12597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:01.884677 12582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:46:01.884775 12582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "57879009e1c6483fb5ed27868f10bd77"
format_stamp: "Formatted at 2026-09-26 20:46:01 on dist-test-slave-4hd3"
server_key: "af6cb938897ced445c4c3a758f5289a6"
server_key_iv: "d62c661587d2b3a773dda83220189946"
server_key_version: "kuduclusterkey@0"
I20260926 20:46:01.884859 12582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:01.897323 12582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:01.899487 12582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:01.899641 12582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:01.904413 12582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:40093
I20260926 20:46:01.904495 12649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:40093 every 8 connection(s)
I20260926 20:46:01.904846 12582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:46:01.905553 12650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:01.906450 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 12582
I20260926 20:46:01.906563 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:46:01.906796 393 external_mini_cluster.cc:1428] Setting key 85469312a356c76e7666105fa578a38c
I20260926 20:46:01.908613 12650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77: Bootstrap starting.
I20260926 20:46:01.909258 12650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:01.909610 12650 log.cc:824] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:01.910377 12650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77: No bootstrap required, opened a new log
I20260926 20:46:01.912755 12650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879009e1c6483fb5ed27868f10bd77" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40093 } }
I20260926 20:46:01.913379 12650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:01.913519 12650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57879009e1c6483fb5ed27868f10bd77, State: Initialized, Role: FOLLOWER
I20260926 20:46:01.913713 12650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879009e1c6483fb5ed27868f10bd77" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40093 } }
I20260926 20:46:01.913821 12650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:46:01.913892 12650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:46:01.913946 12650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:01.914641 12650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879009e1c6483fb5ed27868f10bd77" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40093 } }
I20260926 20:46:01.914749 12650 leader_election.cc:302] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57879009e1c6483fb5ed27868f10bd77; no voters:
Sep 26 20:46:01 dist-test-slave-4hd3 krb5kdc[12138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:01.914948 12650 leader_election.cc:288] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:46:01.915031 12655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:46:01.915196 12655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [term 1 LEADER]: Becoming Leader. State: Replica: 57879009e1c6483fb5ed27868f10bd77, State: Running, Role: LEADER
I20260926 20:46:01.915308 12650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:46:01.915333 12655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879009e1c6483fb5ed27868f10bd77" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40093 } }
I20260926 20:46:01.915777 12655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57879009e1c6483fb5ed27868f10bd77. Latest consensus state: current_term: 1 leader_uuid: "57879009e1c6483fb5ed27868f10bd77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879009e1c6483fb5ed27868f10bd77" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40093 } } }
I20260926 20:46:01.915874 12655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [sys.catalog]: This master's current role is: LEADER
I20260926 20:46:01.915771 12656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57879009e1c6483fb5ed27868f10bd77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879009e1c6483fb5ed27868f10bd77" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40093 } } }
I20260926 20:46:01.916188 12656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [sys.catalog]: This master's current role is: LEADER
I20260926 20:46:01.918519 12650 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 20:46:01.920884 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:01.907291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44454 (local address 127.0.98.126:40093)
0926 20:46:01.907476 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:01.907485 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:01.907939 (+ 454us) server_negotiation.cc:407] Connection header received
0926 20:46:01.908467 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:01.908476 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:01.908534 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:01.908691 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:01.909470 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:01.910789 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:01.911855 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:01.912475 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:01.912592 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:01.915401 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:01.915405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:01.915405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:01.915417 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:01.917413 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:01.917838 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:01.917841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:01.917841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:01.917938 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:01.918189 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:01.918190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:01.918190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:01.918506 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:01.918654 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:01.918814 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 20:46:01.918944 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260926 20:46:01.921558 12659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:46:01.922417 12659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:46:01.924188 12659 catalog_manager.cc:1347] Generated new cluster ID: 02986230a93b4cfca68b0abcd329efeb
I20260926 20:46:01.924240 12659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:46:01.936627 12659 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:46:01.937515 12659 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:46:01.947160 12659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77: Generated new TSK 0
I20260926 20:46:01.947476 12659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260926 20:46:02.683116 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:37999/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40093
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:46:02.792198 12695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:02.792477 12695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:02.792513 12695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:02.793751 12695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:02.793798 12695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:02.793825 12695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:46:02.795348 12695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.84:37999/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:02.795578 12695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:02.795874 12695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:02.800835 12700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:02.800818 12703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:02.801076 12701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:02.801265 12695 server_base.cc:1034] running on GCE node
I20260926 20:46:02.801483 12695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:02.801740 12695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:02.802917 12695 hybrid_clock.cc:648] HybridClock initialized: now 1790455562802897 us; error 34 us; skew 500 ppm
Sep 26 20:46:02 dist-test-slave-4hd3 krb5kdc[12138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:02.805713 12695 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:46:02.806231 12695 webserver.cc:460] Webserver started at http://127.0.98.65:39089/ using document root <none> and password file <none>
I20260926 20:46:02.806478 12695 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:02.806546 12695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:02.806687 12695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:46:02 dist-test-slave-4hd3 krb5kdc[12138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260926 20:46:02.822973 12695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "ca85a2b94e1f463584678e8934fbbee4"
format_stamp: "Formatted at 2026-09-26 20:46:02 on dist-test-slave-4hd3"
server_key: "26949ef469968e0dd3e9e60c5c7658e1"
server_key_iv: "52ba900df6ffe625fd269cf5f0068f0d"
server_key_version: "kuduclusterkey@0"
I20260926 20:46:02.823529 12695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "ca85a2b94e1f463584678e8934fbbee4"
format_stamp: "Formatted at 2026-09-26 20:46:02 on dist-test-slave-4hd3"
server_key: "26949ef469968e0dd3e9e60c5c7658e1"
server_key_iv: "52ba900df6ffe625fd269cf5f0068f0d"
server_key_version: "kuduclusterkey@0"
I20260926 20:46:02.825819 12695 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260926 20:46:02.843869 12712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:02.844120 12695 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260926 20:46:02.844214 12695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "ca85a2b94e1f463584678e8934fbbee4"
format_stamp: "Formatted at 2026-09-26 20:46:02 on dist-test-slave-4hd3"
server_key: "26949ef469968e0dd3e9e60c5c7658e1"
server_key_iv: "52ba900df6ffe625fd269cf5f0068f0d"
server_key_version: "kuduclusterkey@0"
I20260926 20:46:02.844327 12695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:02.876897 12695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:02.879112 12695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:02.879283 12695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:02.879551 12695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:02.879962 12695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:46:02.880007 12695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:02.880097 12695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:46:02.880127 12695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:02.886994 12695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:44043
I20260926 20:46:02.887142 12825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:44043 every 8 connection(s)
I20260926 20:46:02.887497 12695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:46:02.890197 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 12695
I20260926 20:46:02.890306 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:46:02.890493 393 external_mini_cluster.cc:1428] Setting key 0cbeb4de43bca427f9c3cc26765c72cb
Sep 26 20:46:02 dist-test-slave-4hd3 krb5kdc[12138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:02.898944 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:02.888320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40093 (local address 127.0.98.65:45933)
0926 20:46:02.888542 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:02.888583 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:02.889246 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:02.889536 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:02.889540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:02.889789 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:02.890012 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:02.890019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:02.890852 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:02.890853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:02.891866 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:02.891869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:02.892450 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:02.892451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:02.892589 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:02.893206 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:02.893216 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:02.894808 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:02.897074 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:02.897081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:02.897084 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:02.897440 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:02.897688 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:02.897689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:02.897690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:02.897840 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:02.898146 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:02.898151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:02.898343 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 20:46:02.898472 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 20:46:02.898654 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
I20260926 20:46:02.899595 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:02.888320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:45933 (local address 127.0.98.126:40093)
0926 20:46:02.888478 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:02.888480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:02.888702 (+ 222us) server_negotiation.cc:407] Connection header received
0926 20:46:02.889354 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:02.889355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:02.889371 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:02.889442 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:02.890102 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:02.890751 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:02.891961 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:02.892296 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:02.892361 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:02.894916 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:02.894918 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:02.894918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:02.894926 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:02.896969 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:02.897533 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:02.897534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:02.897535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:02.897607 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:02.897911 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:02.897912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:02.897912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:02.898058 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:02.898119 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:02.899335 (+ 1216us) server_negotiation.cc:299] Negotiation successful
0926 20:46:02.899449 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260926 20:46:02.900419 12826 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40093
I20260926 20:46:02.900520 12826 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:02.900714 12826 heartbeater.cc:511] Master 127.0.98.126:40093 requested a full tablet report, sending...
I20260926 20:46:02.901321 12614 ts_manager.cc:194] Registered new tserver with Master: ca85a2b94e1f463584678e8934fbbee4 (127.0.98.65:44043)
I20260926 20:46:02.902279 12614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:45933
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260926 20:46:02.939024 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:37999/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40093
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:46:03 dist-test-slave-4hd3 krb5kdc[12138](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790455563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 20:46:03.036278 12834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:03.036504 12834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:03.036544 12834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:03.037863 12834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:03.037964 12834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:03.038026 12834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:46:03.040233 12834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.84:37999/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:03.040585 12834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:03.040866 12834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:03.046717 12839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:03.046682 12840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:03.046818 12842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:03.047945 12834 server_base.cc:1034] running on GCE node
I20260926 20:46:03.048162 12834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:03.048425 12834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:03.049597 12834 hybrid_clock.cc:648] HybridClock initialized: now 1790455563049564 us; error 47 us; skew 500 ppm
Sep 26 20:46:03 dist-test-slave-4hd3 krb5kdc[12138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:03.053068 12834 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:46:03.053637 12834 webserver.cc:460] Webserver started at http://127.0.98.66:45599/ using document root <none> and password file <none>
I20260926 20:46:03.053982 12834 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:03.054096 12834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:03.054267 12834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:46:03 dist-test-slave-4hd3 krb5kdc[12138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260926 20:46:03.074779 12834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "95ac889929fb4ebea6bd466bfdf4faed"
format_stamp: "Formatted at 2026-09-26 20:46:03 on dist-test-slave-4hd3"
server_key: "037bb6126efa50efef0575e7d08cc4ea"
server_key_iv: "6b40cb06b26f7aee7a679f26f83fb470"
server_key_version: "kuduclusterkey@0"
I20260926 20:46:03.075289 12834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "95ac889929fb4ebea6bd466bfdf4faed"
format_stamp: "Formatted at 2026-09-26 20:46:03 on dist-test-slave-4hd3"
server_key: "037bb6126efa50efef0575e7d08cc4ea"
server_key_iv: "6b40cb06b26f7aee7a679f26f83fb470"
server_key_version: "kuduclusterkey@0"
I20260926 20:46:03.079844 12834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20260926 20:46:03.093902 12849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:03.094098 12834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:46:03.094187 12834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "95ac889929fb4ebea6bd466bfdf4faed"
format_stamp: "Formatted at 2026-09-26 20:46:03 on dist-test-slave-4hd3"
server_key: "037bb6126efa50efef0575e7d08cc4ea"
server_key_iv: "6b40cb06b26f7aee7a679f26f83fb470"
server_key_version: "kuduclusterkey@0"
I20260926 20:46:03.094293 12834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:03.134680 12834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:03.137356 12834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:03.137564 12834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:03.137926 12834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:03.138450 12834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:46:03.138540 12834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:03.138622 12834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:46:03.138684 12834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:03.146118 12834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:41351
I20260926 20:46:03.146669 12834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:46:03.148407 12963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:41351 every 8 connection(s)
I20260926 20:46:03.154772 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 12834
I20260926 20:46:03.154886 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:46:03.155082 393 external_mini_cluster.cc:1428] Setting key 29519c3844d07ac5c52f5fcdfaa6eec0
Sep 26 20:46:03 dist-test-slave-4hd3 krb5kdc[12138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:03.160912 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:03.149875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:55599 (local address 127.0.98.126:40093)
0926 20:46:03.149920 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:03.149922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:03.150287 (+ 365us) server_negotiation.cc:407] Connection header received
0926 20:46:03.151007 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:03.151009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:03.151022 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:03.151052 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:03.151972 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:03.152744 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:03.154293 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:03.154598 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:03.154649 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:03.157412 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:03.157414 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:03.157414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:03.157422 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:03.159223 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:03.159740 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:03.159741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:03.159741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:03.159800 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:03.160101 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:03.160102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:03.160102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:03.160236 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:03.160277 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:03.160687 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 20:46:03.160780 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:46:03.161724 12966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:03.149572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40093 (local address 127.0.98.66:55599)
0926 20:46:03.150148 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:03.150198 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:03.150874 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:03.151239 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:03.151244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:03.151506 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:03.151857 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:03.151865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:03.152912 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:03.152913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:03.154186 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:03.154189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:03.154758 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:03.154759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:03.154908 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:03.155555 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:03.155565 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:03.157295 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:03.159326 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:03.159333 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:03.159336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:03.159631 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:03.159886 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:03.159887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:03.159887 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:03.160026 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:03.160323 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:03.160330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:03.160527 (+ 197us) client_negotiation.cc:769] Sending connection context
0926 20:46:03.161318 (+ 791us) client_negotiation.cc:240] Negotiation successful
0926 20:46:03.161491 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":90,"threads_started":1}
I20260926 20:46:03.162398 12964 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40093
I20260926 20:46:03.162501 12964 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:03.162693 12964 heartbeater.cc:511] Master 127.0.98.126:40093 requested a full tablet report, sending...
I20260926 20:46:03.163125 12614 ts_manager.cc:194] Registered new tserver with Master: 95ac889929fb4ebea6bd466bfdf4faed (127.0.98.66:41351)
I20260926 20:46:03.163605 12614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:55599
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260926 20:46:03.194792 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:37999/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40093
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:46:03.319141 12969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:03.319504 12969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:03.319583 12969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:03.321323 12969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:03.321424 12969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:03.321507 12969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:46:03.323634 12969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.84:37999/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:03.324051 12969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:03.324352 12969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:03.330547 12975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:03.330699 12974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:03.330909 12977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:03.330945 12969 server_base.cc:1034] running on GCE node
I20260926 20:46:03.331141 12969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:03.331393 12969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:03.332544 12969 hybrid_clock.cc:648] HybridClock initialized: now 1790455563332517 us; error 42 us; skew 500 ppm
Sep 26 20:46:03 dist-test-slave-4hd3 krb5kdc[12138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:03.335834 12969 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:46:03.336360 12969 webserver.cc:460] Webserver started at http://127.0.98.67:43935/ using document root <none> and password file <none>
I20260926 20:46:03.336602 12969 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:03.336658 12969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:03.336783 12969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:46:03 dist-test-slave-4hd3 krb5kdc[12138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260926 20:46:03.365980 12969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "5b459f2625914997aeefdaa46dab79b7"
format_stamp: "Formatted at 2026-09-26 20:46:03 on dist-test-slave-4hd3"
server_key: "7da75396f24d1cd6958776ad05af024e"
server_key_iv: "9c35cf24de3b5f286fcbeb556cc1a18c"
server_key_version: "kuduclusterkey@0"
I20260926 20:46:03.366403 12969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "5b459f2625914997aeefdaa46dab79b7"
format_stamp: "Formatted at 2026-09-26 20:46:03 on dist-test-slave-4hd3"
server_key: "7da75396f24d1cd6958776ad05af024e"
server_key_iv: "9c35cf24de3b5f286fcbeb556cc1a18c"
server_key_version: "kuduclusterkey@0"
I20260926 20:46:03.367954 12969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:46:03.399844 12984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:03.400054 12969 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260926 20:46:03.400146 12969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "5b459f2625914997aeefdaa46dab79b7"
format_stamp: "Formatted at 2026-09-26 20:46:03 on dist-test-slave-4hd3"
server_key: "7da75396f24d1cd6958776ad05af024e"
server_key_iv: "9c35cf24de3b5f286fcbeb556cc1a18c"
server_key_version: "kuduclusterkey@0"
I20260926 20:46:03.400254 12969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:03.412688 12969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:03.415115 12969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:03.415385 12969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:03.415742 12969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:03.416152 12969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:46:03.416245 12969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:03.416327 12969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:46:03.416404 12969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:03.423336 12969 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:45149
I20260926 20:46:03.423873 12969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:46:03.425527 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 12969
I20260926 20:46:03.425638 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:46:03.425836 393 external_mini_cluster.cc:1428] Setting key 578d79bcd86736fcbfad5c872f852864
I20260926 20:46:03.427667 13097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:45149 every 8 connection(s)
Sep 26 20:46:03 dist-test-slave-4hd3 krb5kdc[12138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:03.445502 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:03.428885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:50507 (local address 127.0.98.126:40093)
0926 20:46:03.428926 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:03.428929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:03.429249 (+ 320us) server_negotiation.cc:407] Connection header received
0926 20:46:03.429947 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:03.429948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:03.429960 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:03.429990 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:03.430870 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:03.431641 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:03.433108 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:03.433387 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:03.433442 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:03.440308 (+ 6866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:03.440310 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:03.440310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:03.440319 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:03.442147 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:03.442848 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:03.442849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:03.442850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:03.442914 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:03.443238 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:03.443239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:03.443240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:03.443380 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:03.443441 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:03.445293 (+ 1852us) server_negotiation.cc:299] Negotiation successful
0926 20:46:03.445384 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:46:03.446765 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:03.428599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40093 (local address 127.0.98.67:50507)
0926 20:46:03.429123 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:03.429167 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:03.429821 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:03.430178 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:03.430182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:03.430445 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:03.430727 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:03.430733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:03.431797 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:03.431797 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:03.432999 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:03.433003 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:03.433546 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:03.433547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:03.433680 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:03.437881 (+ 4201us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:03.437893 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:03.440189 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:03.442422 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:03.442430 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:03.442433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:03.442750 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:03.442998 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:03.442999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:03.442999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:03.443150 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:03.443495 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:03.443502 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:03.443691 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 20:46:03.446311 (+ 2620us) client_negotiation.cc:240] Negotiation successful
0926 20:46:03.446517 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":96,"threads_started":1}
I20260926 20:46:03.447476 13098 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40093
I20260926 20:46:03.447573 13098 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:03.447815 13098 heartbeater.cc:511] Master 127.0.98.126:40093 requested a full tablet report, sending...
I20260926 20:46:03.448875 12614 ts_manager.cc:194] Registered new tserver with Master: 5b459f2625914997aeefdaa46dab79b7 (127.0.98.67:45149)
I20260926 20:46:03.449371 12614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:50507
I20260926 20:46:03.459416 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:46:03.460362 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 12695
I20260926 20:46:03.468317 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 12834
I20260926 20:46:03.474706 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 12969
I20260926 20:46:03.482288 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 12582
I20260926 20:46:03.488725 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40093
--webserver_interface=127.0.98.126
--webserver_port=41239
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:40093
--ranger_config_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:37999/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:37999/kms with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:46:03.571135 13101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:03.571386 13101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:03.571424 13101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:03.572597 13101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:46:03.572666 13101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:03.572695 13101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:03.572726 13101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:46:03.572749 13101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:46:03.574404 13101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:37999/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:40093
--ranger_config_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=41239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:03.574676 13101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:03.574908 13101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:03.579559 13106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:03.579682 13107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:03.579564 13109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:03.580001 13101 server_base.cc:1034] running on GCE node
I20260926 20:46:03.580292 13101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:03.580616 13101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:03.581800 13101 hybrid_clock.cc:648] HybridClock initialized: now 1790455563581786 us; error 28 us; skew 500 ppm
Sep 26 20:46:03 dist-test-slave-4hd3 krb5kdc[12138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:03.584851 13101 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:46:03.585333 13101 webserver.cc:460] Webserver started at http://127.0.98.126:41239/ using document root <none> and password file <none>
I20260926 20:46:03.585564 13101 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:03.585641 13101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:46:03 dist-test-slave-4hd3 krb5kdc[12138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260926 20:46:03.649667 13101 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:46:03.650511 13117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:03.650717 13101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:46:03.650820 13101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "57879009e1c6483fb5ed27868f10bd77"
format_stamp: "Formatted at 2026-09-26 20:46:01 on dist-test-slave-4hd3"
server_key: "af6cb938897ced445c4c3a758f5289a6"
server_key_iv: "d62c661587d2b3a773dda83220189946"
server_key_version: "kuduclusterkey@0"
I20260926 20:46:03.651286 13101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:03.698532 13101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:03.700417 13101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:03.700541 13101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:03.704798 13101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:40093
I20260926 20:46:03.704842 13169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:40093 every 8 connection(s)
I20260926 20:46:03.705204 13101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:46:03.705878 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 13101
I20260926 20:46:03.706149 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:37999/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:44043
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:40093
--webserver_port=39089
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:37999/kms with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:46:03.706617 13170 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:46:03.707288 13170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77: Bootstrap starting.
I20260926 20:46:03.715353 13170 log.cc:824] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:03.717921 13170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:46:03.718266 13170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77: Bootstrap complete.
I20260926 20:46:03.721164 13170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879009e1c6483fb5ed27868f10bd77" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40093 } }
I20260926 20:46:03.721412 13170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57879009e1c6483fb5ed27868f10bd77, State: Initialized, Role: FOLLOWER
I20260926 20:46:03.721585 13170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879009e1c6483fb5ed27868f10bd77" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40093 } }
I20260926 20:46:03.721661 13170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:46:03.721728 13170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:46:03.721769 13170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:46:03.722790 13170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879009e1c6483fb5ed27868f10bd77" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40093 } }
I20260926 20:46:03.722874 13170 leader_election.cc:302] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57879009e1c6483fb5ed27868f10bd77; no voters:
I20260926 20:46:03.723075 13170 leader_election.cc:288] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:46:03.723135 13174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:46:03.723268 13174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [term 2 LEADER]: Becoming Leader. State: Replica: 57879009e1c6483fb5ed27868f10bd77, State: Running, Role: LEADER
I20260926 20:46:03.723351 13170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:46:03.723357 13174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879009e1c6483fb5ed27868f10bd77" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40093 } }
I20260926 20:46:03.723614 13174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57879009e1c6483fb5ed27868f10bd77. Latest consensus state: current_term: 2 leader_uuid: "57879009e1c6483fb5ed27868f10bd77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879009e1c6483fb5ed27868f10bd77" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40093 } } }
I20260926 20:46:03.723701 13174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [sys.catalog]: This master's current role is: LEADER
I20260926 20:46:03.723929 13178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:46:03.723616 13175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "57879009e1c6483fb5ed27868f10bd77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57879009e1c6483fb5ed27868f10bd77" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40093 } } }
I20260926 20:46:03.724136 13175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77 [sys.catalog]: This master's current role is: LEADER
I20260926 20:46:03.724397 13178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:46:03.724517 13178 catalog_manager.cc:1259] Loaded cluster ID: 02986230a93b4cfca68b0abcd329efeb
I20260926 20:46:03.724551 13178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:46:03.724970 13170 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 20:46:03.728029 13178 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:46:03.728742 13178 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77: Loaded TSK: 0
I20260926 20:46:03.729135 13178 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:46:03.821823 13172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:03.822074 13172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:03.822108 13172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:03.823943 13172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:03.823997 13172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:03.824043 13172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:46:03.826270 13172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:37999/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:44043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=39089
--tserver_master_addrs=127.0.98.126:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:03.826509 13172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:03.826753 13172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:03.833331 13208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:03.833911 13210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:03.834067 13207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:03.838471 13172 server_base.cc:1034] running on GCE node
I20260926 20:46:03.838692 13172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:03.838963 13172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:03.840128 13172 hybrid_clock.cc:648] HybridClock initialized: now 1790455563840091 us; error 51 us; skew 500 ppm
Sep 26 20:46:03 dist-test-slave-4hd3 krb5kdc[12138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:03.843429 13172 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:46:03.843959 13172 webserver.cc:460] Webserver started at http://127.0.98.65:39089/ using document root <none> and password file <none>
I20260926 20:46:03.844231 13172 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:03.844319 13172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:46:03 dist-test-slave-4hd3 krb5kdc[12138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260926 20:46:03.867054 13172 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.001s
I20260926 20:46:03.867858 13218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:03.868014 13172 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:46:03.868096 13172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "ca85a2b94e1f463584678e8934fbbee4"
format_stamp: "Formatted at 2026-09-26 20:46:02 on dist-test-slave-4hd3"
server_key: "26949ef469968e0dd3e9e60c5c7658e1"
server_key_iv: "52ba900df6ffe625fd269cf5f0068f0d"
server_key_version: "kuduclusterkey@0"
I20260926 20:46:03.869153 13172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:03.885771 13172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:03.888048 13172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:03.888200 13172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:03.888427 13172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:03.888783 13172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:46:03.888821 13172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:03.888850 13172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:46:03.888870 13172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:03.895951 13172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:44043
I20260926 20:46:03.896450 13172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:46:03.896756 13331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:44043 every 8 connection(s)
I20260926 20:46:03.902803 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 13172
I20260926 20:46:03.903113 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:37999/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:41351
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:40093
--webserver_port=45599
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:37999/kms with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:46:03 dist-test-slave-4hd3 krb5kdc[12138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:03.908360 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:03.897264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40093 (local address 127.0.98.65:46057)
0926 20:46:03.897439 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:03.897494 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:03.898079 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:03.898576 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:03.898580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:03.898816 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:03.899042 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:03.899047 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:03.900215 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:03.900217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:03.901021 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:03.901023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:03.901730 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:03.901731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:03.901858 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:03.902411 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:03.902419 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:03.904159 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:03.906361 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:03.906367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:03.906369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:03.906687 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:03.907005 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:03.907006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:03.907006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:03.907152 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:03.907654 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:03.907660 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:03.907843 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 20:46:03.907965 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 20:46:03.908129 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260926 20:46:03.908456 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:03.897398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:46057 (local address 127.0.98.126:40093)
0926 20:46:03.897613 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:03.897623 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:03.897666 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:46:03.898324 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:03.898330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:03.898392 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:03.898537 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:03.899154 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:03.900112 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:03.901123 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:03.901620 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:03.901744 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:03.904357 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:03.904362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:03.904363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:03.904378 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:03.906277 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:03.906788 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:03.906791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:03.906792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:03.906916 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:03.907225 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:03.907225 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:03.907226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:03.907537 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:03.907623 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:03.908059 (+ 436us) server_negotiation.cc:299] Negotiation successful
0926 20:46:03.908201 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":103,"threads_started":1}
I20260926 20:46:03.909936 13332 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40093
I20260926 20:46:03.910034 13332 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:03.910215 13332 heartbeater.cc:511] Master 127.0.98.126:40093 requested a full tablet report, sending...
W20260926 20:46:03.911173 13332 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:40093 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 20:46:03.911250 13332 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:03.911315 13332 heartbeater.cc:511] Master 127.0.98.126:40093 requested a full tablet report, sending...
I20260926 20:46:03.911681 13332 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:03.911784 13332 heartbeater.cc:511] Master 127.0.98.126:40093 requested a full tablet report, sending...
W20260926 20:46:03.912122 13332 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:46:04.001354 13336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:04.001561 13336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:04.001585 13336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:04.002835 13336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:04.002878 13336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:04.002907 13336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:46:04.004552 13336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:37999/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:41351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=45599
--tserver_master_addrs=127.0.98.126:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:04.004817 13336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:04.005082 13336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:04.011636 13342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:04.011651 13336 server_base.cc:1034] running on GCE node
W20260926 20:46:04.011790 13341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:04.011638 13344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:04.012149 13336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:04.012429 13336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:04.013624 13336 hybrid_clock.cc:648] HybridClock initialized: now 1790455564013589 us; error 50 us; skew 500 ppm
Sep 26 20:46:04 dist-test-slave-4hd3 krb5kdc[12138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:04.016892 13336 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:46:04.017468 13336 webserver.cc:460] Webserver started at http://127.0.98.66:45599/ using document root <none> and password file <none>
I20260926 20:46:04.017714 13336 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:04.017782 13336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:46:04 dist-test-slave-4hd3 krb5kdc[12138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260926 20:46:04.034387 13336 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:46:04.035435 13352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:04.035681 13336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:46:04.035846 13336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "95ac889929fb4ebea6bd466bfdf4faed"
format_stamp: "Formatted at 2026-09-26 20:46:03 on dist-test-slave-4hd3"
server_key: "037bb6126efa50efef0575e7d08cc4ea"
server_key_iv: "6b40cb06b26f7aee7a679f26f83fb470"
server_key_version: "kuduclusterkey@0"
I20260926 20:46:04.036350 13336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:04.049636 13336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:04.051887 13336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:04.052059 13336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:04.052336 13336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:04.052690 13336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:46:04.052743 13336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:04.052778 13336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:46:04.052806 13336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:04.059983 13336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:41351
I20260926 20:46:04.060457 13336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:46:04.061072 13465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:41351 every 8 connection(s)
Sep 26 20:46:04 dist-test-slave-4hd3 krb5kdc[12138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:04.070016 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 13336
I20260926 20:46:04.070328 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:37999/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:45149
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:40093
--webserver_port=43935
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:37999/kms with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:46:04.071996 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:04.061566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:57421 (local address 127.0.98.126:40093)
0926 20:46:04.061646 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:04.061648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:04.061660 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:46:04.062219 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:04.062221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:04.062233 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:04.062305 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:04.062974 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:04.063775 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:04.065259 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:04.065581 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:04.065664 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:04.068304 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:04.068306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:04.068306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:04.068316 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:04.070210 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:04.070749 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:04.070750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:04.070751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:04.070828 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:04.071120 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:04.071121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:04.071121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:04.071265 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:04.071309 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:04.071708 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 20:46:04.071850 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:46:04.072633 13468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:04.061238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40093 (local address 127.0.98.66:57421)
0926 20:46:04.061435 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:04.061483 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:04.062100 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:04.062374 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:04.062379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:04.062597 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:04.062862 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:04.062868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:04.063892 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:04.063893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:04.065160 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:04.065163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:04.065706 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:04.065708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:04.065851 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:04.066491 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:04.066501 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:04.068189 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:04.070314 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:04.070322 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:04.070324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:04.070650 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:04.070905 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:04.070905 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:04.070906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:04.071040 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:04.071360 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:04.071366 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:04.071539 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 20:46:04.072140 (+ 601us) client_negotiation.cc:240] Negotiation successful
0926 20:46:04.072324 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":95,"threads_started":1}
I20260926 20:46:04.073119 13466 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40093
I20260926 20:46:04.073272 13466 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:04.073513 13466 heartbeater.cc:511] Master 127.0.98.126:40093 requested a full tablet report, sending...
W20260926 20:46:04.074186 13466 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:40093 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 20:46:04.074337 13466 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:04.074455 13466 heartbeater.cc:511] Master 127.0.98.126:40093 requested a full tablet report, sending...
I20260926 20:46:04.074800 13466 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:04.074919 13466 heartbeater.cc:511] Master 127.0.98.126:40093 requested a full tablet report, sending...
W20260926 20:46:04.075243 13466 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:46:04.171001 13469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:04.171226 13469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:04.171264 13469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:04.172540 13469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:04.172601 13469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:04.172646 13469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:46:04.174451 13469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:37999/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:45149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=43935
--tserver_master_addrs=127.0.98.126:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:04.174718 13469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:04.174996 13469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:04.181353 13475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:04.181640 13477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:04.182792 13469 server_base.cc:1034] running on GCE node
W20260926 20:46:04.183579 13474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:04.183872 13469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:04.184198 13469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:04.185452 13469 hybrid_clock.cc:648] HybridClock initialized: now 1790455564185432 us; error 33 us; skew 500 ppm
Sep 26 20:46:04 dist-test-slave-4hd3 krb5kdc[12138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:04.188573 13469 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:46:04.189128 13469 webserver.cc:460] Webserver started at http://127.0.98.67:43935/ using document root <none> and password file <none>
I20260926 20:46:04.189416 13469 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:04.189502 13469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:46:04 dist-test-slave-4hd3 krb5kdc[12138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260926 20:46:04.220137 13469 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.001s sys 0.000s
I20260926 20:46:04.221006 13485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:04.221210 13469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:46:04.221311 13469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "5b459f2625914997aeefdaa46dab79b7"
format_stamp: "Formatted at 2026-09-26 20:46:03 on dist-test-slave-4hd3"
server_key: "7da75396f24d1cd6958776ad05af024e"
server_key_iv: "9c35cf24de3b5f286fcbeb556cc1a18c"
server_key_version: "kuduclusterkey@0"
I20260926 20:46:04.221760 13469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:04.246111 13469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:04.248543 13469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:04.248703 13469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:04.248983 13469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:04.249349 13469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:46:04.249387 13469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:04.249409 13469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:46:04.249423 13469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:04.258283 13469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:45149
I20260926 20:46:04.258765 13469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:46:04.259274 13599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:45149 every 8 connection(s)
I20260926 20:46:04.266125 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 13469
Sep 26 20:46:04 dist-test-slave-4hd3 krb5kdc[12138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:04.270969 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:04.260675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:44123 (local address 127.0.98.126:40093)
0926 20:46:04.260723 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:04.260725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:04.260980 (+ 255us) server_negotiation.cc:407] Connection header received
0926 20:46:04.261679 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:04.261681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:04.261694 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:04.261725 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:04.262550 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:04.263302 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:04.264715 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:04.264934 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:04.265023 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:04.267526 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:04.267527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:04.267528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:04.267535 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:04.269239 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:04.269774 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:04.269775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:04.269776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:04.269841 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:04.270167 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:04.270168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:04.270168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:04.270338 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:04.270409 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:04.270772 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 20:46:04.270862 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:46:04.271157 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:04.260382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40093 (local address 127.0.98.67:44123)
0926 20:46:04.260842 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:04.260890 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:04.261549 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:04.261929 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:04.261934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:04.262174 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:04.262456 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:04.262464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:04.263406 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:04.263408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:04.264621 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:04.264624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:04.265053 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:04.265055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:04.265187 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:04.265784 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:04.265804 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:04.267431 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:04.269343 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:04.269350 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:04.269353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:04.269688 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:04.269924 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:04.269926 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:04.269926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:04.270083 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:04.270440 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:04.270447 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:04.270623 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 20:46:04.270748 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 20:46:04.270912 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":90,"threads_started":1}
I20260926 20:46:04.271806 13600 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40093
I20260926 20:46:04.271903 13600 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:04.272053 13600 heartbeater.cc:511] Master 127.0.98.126:40093 requested a full tablet report, sending...
W20260926 20:46:04.272751 13600 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:40093 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 20:46:04.272814 13600 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:04.272933 13600 heartbeater.cc:511] Master 127.0.98.126:40093 requested a full tablet report, sending...
I20260926 20:46:04.273490 13600 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:04.273581 13600 heartbeater.cc:511] Master 127.0.98.126:40093 requested a full tablet report, sending...
W20260926 20:46:04.273905 13600 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:46:04.276463 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:04.266646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59736 (local address 127.0.98.126:40093)
0926 20:46:04.266813 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:04.266815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:04.266897 (+ 82us) server_negotiation.cc:407] Connection header received
0926 20:46:04.266952 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:04.266953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:04.266966 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:04.267050 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:04.268222 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:04.268999 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:04.270183 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:04.270488 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:04.270558 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:04.273782 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:04.273783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:04.273784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:04.273791 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:04.275367 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:04.275695 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:04.275696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:04.275696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:04.275774 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:04.275927 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:04.275928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:04.275928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:04.276047 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:04.276212 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:04.276282 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 20:46:04.276364 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":68,"threads_started":1}
Sep 26 20:46:04 dist-test-slave-4hd3 krb5kdc[12138](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790455564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:04.912292 13332 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:04.912448 13332 heartbeater.cc:511] Master 127.0.98.126:40093 requested a full tablet report, sending...
I20260926 20:46:04.912895 13134 ts_manager.cc:194] Registered new tserver with Master: ca85a2b94e1f463584678e8934fbbee4 (127.0.98.65:44043)
I20260926 20:46:04.913950 13134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:46057
I20260926 20:46:05.075465 13466 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:05.075626 13466 heartbeater.cc:511] Master 127.0.98.126:40093 requested a full tablet report, sending...
I20260926 20:46:05.076009 13134 ts_manager.cc:194] Registered new tserver with Master: 95ac889929fb4ebea6bd466bfdf4faed (127.0.98.66:41351)
I20260926 20:46:05.076537 13134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:57421
Sep 26 20:46:05 dist-test-slave-4hd3 krb5kdc[12138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260926 20:46:05.274086 13600 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:05.274238 13600 heartbeater.cc:511] Master 127.0.98.126:40093 requested a full tablet report, sending...
I20260926 20:46:05.274524 13134 ts_manager.cc:194] Registered new tserver with Master: 5b459f2625914997aeefdaa46dab79b7 (127.0.98.67:45149)
I20260926 20:46:05.275043 13134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:44123
I20260926 20:46:05.283183 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:46:05.292356 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:05.284209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59740 (local address 127.0.98.126:40093)
0926 20:46:05.284614 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:05.284616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:05.284626 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:46:05.284651 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:05.284651 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:05.284663 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:05.284735 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:05.285482 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.286304 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.287342 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.287651 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.287757 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:05.289274 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:05.289275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:05.289276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:05.289283 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:05.291061 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:05.291469 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:05.291470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:05.291471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:05.291534 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:05.291795 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:05.291796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:05.291796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:05.291922 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:05.292020 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:05.292162 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 20:46:05.292244 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":68,"threads_started":1}
I20260926 20:46:05.294673 13134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:46:05.296818 13134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:46:05.304772 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:05.299658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:45149 (local address 127.0.0.1:46134)
0926 20:46:05.299823 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:05.299865 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:05.299935 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:05.300802 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:05.300806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:05.300808 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:05.300892 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:05.300894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.302191 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.302192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:05.303488 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:05.303491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.304485 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.304486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:05.304568 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:05.304584 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 20:46:05.304634 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 20:46:05.304694 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260926 20:46:05.305249 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:05.299876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:44043 (local address 127.0.0.1:32940)
0926 20:46:05.300868 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:05.300878 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:05.300922 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:05.301167 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:05.301168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:05.301170 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:05.301210 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:05.301211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.302614 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.302615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:05.303678 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:05.303681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.304961 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.304962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:05.305018 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:05.305021 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:46:05.305145 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 20:46:05.305182 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"thread_start_us":53,"threads_started":1}
I20260926 20:46:05.305475 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:05.300208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46134 (local address 127.0.98.67:45149)
0926 20:46:05.300622 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:05.300627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:05.300636 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:46:05.300669 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:05.300671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:05.300686 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:05.300766 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:05.300992 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.302092 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.303592 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.304378 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.305087 (+ 709us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:05.305150 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:05.305177 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 20:46:05.305216 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":54,"threads_started":1}
I20260926 20:46:05.305508 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:05.299986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32940 (local address 127.0.98.65:44043)
0926 20:46:05.300096 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:05.300101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:05.300983 (+ 882us) server_negotiation.cc:407] Connection header received
0926 20:46:05.301021 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:05.301023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:05.301037 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:05.301096 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:05.301270 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.302533 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.303824 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.304821 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.304878 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:05.304930 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:05.305400 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 20:46:05.305439 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":58,"threads_started":1}
I20260926 20:46:05.306182 13254 tablet_service.cc:1467] Processing CreateTablet for tablet 55a0d14968c8428bb9326574720ea59d (DEFAULT_TABLE table=test-table [id=cd388c6fd0db4f47852d513f4c4e2a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:46:05.306241 13534 tablet_service.cc:1467] Processing CreateTablet for tablet 55a0d14968c8428bb9326574720ea59d (DEFAULT_TABLE table=test-table [id=cd388c6fd0db4f47852d513f4c4e2a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:46:05.306529 13534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55a0d14968c8428bb9326574720ea59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:05.306530 13254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55a0d14968c8428bb9326574720ea59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:05.309947 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:05.300018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:41351 (local address 127.0.0.1:40132)
0926 20:46:05.303776 (+ 3758us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:05.303786 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:05.303831 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:05.306186 (+ 2355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:05.306188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:05.306189 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:05.306222 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:05.306224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.307669 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.307670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:05.308760 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:05.308763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.309730 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.309731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:05.309789 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:05.309793 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:46:05.309838 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 20:46:05.309874 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3740,"thread_start_us":60,"threads_started":1}
I20260926 20:46:05.310321 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:05.300187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40132 (local address 127.0.98.66:41351)
0926 20:46:05.305974 (+ 5787us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:05.305978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:05.305988 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:46:05.306020 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:05.306022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:05.306036 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:05.306108 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:05.306286 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.307589 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.308858 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.309579 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.309647 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:05.309696 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:05.310203 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 20:46:05.310246 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5750,"thread_start_us":71,"threads_started":1}
I20260926 20:46:05.310930 13400 tablet_service.cc:1467] Processing CreateTablet for tablet 55a0d14968c8428bb9326574720ea59d (DEFAULT_TABLE table=test-table [id=cd388c6fd0db4f47852d513f4c4e2a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:46:05.311239 13400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55a0d14968c8428bb9326574720ea59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:05.314307 13634 tablet_bootstrap.cc:492] T 55a0d14968c8428bb9326574720ea59d P ca85a2b94e1f463584678e8934fbbee4: Bootstrap starting.
I20260926 20:46:05.314976 13634 tablet_bootstrap.cc:654] T 55a0d14968c8428bb9326574720ea59d P ca85a2b94e1f463584678e8934fbbee4: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:05.315299 13634 log.cc:824] T 55a0d14968c8428bb9326574720ea59d P ca85a2b94e1f463584678e8934fbbee4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:05.315637 13633 tablet_bootstrap.cc:492] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7: Bootstrap starting.
I20260926 20:46:05.316174 13634 tablet_bootstrap.cc:492] T 55a0d14968c8428bb9326574720ea59d P ca85a2b94e1f463584678e8934fbbee4: No bootstrap required, opened a new log
I20260926 20:46:05.316254 13633 tablet_bootstrap.cc:654] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:05.316354 13634 ts_tablet_manager.cc:1397] T 55a0d14968c8428bb9326574720ea59d P ca85a2b94e1f463584678e8934fbbee4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:46:05.316571 13633 log.cc:824] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:05.317291 13633 tablet_bootstrap.cc:492] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7: No bootstrap required, opened a new log
I20260926 20:46:05.317370 13633 ts_tablet_manager.cc:1397] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:46:05.318737 13634 raft_consensus.cc:348] T 55a0d14968c8428bb9326574720ea59d P ca85a2b94e1f463584678e8934fbbee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b459f2625914997aeefdaa46dab79b7" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45149 } } peers { permanent_uuid: "95ac889929fb4ebea6bd466bfdf4faed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41351 } } peers { permanent_uuid: "ca85a2b94e1f463584678e8934fbbee4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44043 } }
I20260926 20:46:05.319312 13634 raft_consensus.cc:374] T 55a0d14968c8428bb9326574720ea59d P ca85a2b94e1f463584678e8934fbbee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:05.319415 13634 raft_consensus.cc:729] T 55a0d14968c8428bb9326574720ea59d P ca85a2b94e1f463584678e8934fbbee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca85a2b94e1f463584678e8934fbbee4, State: Initialized, Role: FOLLOWER
I20260926 20:46:05.319592 13634 consensus_queue.cc:260] T 55a0d14968c8428bb9326574720ea59d P ca85a2b94e1f463584678e8934fbbee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b459f2625914997aeefdaa46dab79b7" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45149 } } peers { permanent_uuid: "95ac889929fb4ebea6bd466bfdf4faed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41351 } } peers { permanent_uuid: "ca85a2b94e1f463584678e8934fbbee4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44043 } }
I20260926 20:46:05.319641 13633 raft_consensus.cc:348] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b459f2625914997aeefdaa46dab79b7" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45149 } } peers { permanent_uuid: "95ac889929fb4ebea6bd466bfdf4faed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41351 } } peers { permanent_uuid: "ca85a2b94e1f463584678e8934fbbee4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44043 } }
I20260926 20:46:05.319829 13633 raft_consensus.cc:374] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:05.319870 13633 raft_consensus.cc:729] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b459f2625914997aeefdaa46dab79b7, State: Initialized, Role: FOLLOWER
I20260926 20:46:05.319854 13635 tablet_bootstrap.cc:492] T 55a0d14968c8428bb9326574720ea59d P 95ac889929fb4ebea6bd466bfdf4faed: Bootstrap starting.
I20260926 20:46:05.319964 13633 consensus_queue.cc:260] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b459f2625914997aeefdaa46dab79b7" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45149 } } peers { permanent_uuid: "95ac889929fb4ebea6bd466bfdf4faed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41351 } } peers { permanent_uuid: "ca85a2b94e1f463584678e8934fbbee4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44043 } }
I20260926 20:46:05.320194 13633 ts_tablet_manager.cc:1428] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:46:05.320195 13634 ts_tablet_manager.cc:1428] T 55a0d14968c8428bb9326574720ea59d P ca85a2b94e1f463584678e8934fbbee4: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 20:46:05.320320 13600 heartbeater.cc:503] Master 127.0.98.126:40093 was elected leader, sending a full tablet report...
I20260926 20:46:05.320355 13332 heartbeater.cc:503] Master 127.0.98.126:40093 was elected leader, sending a full tablet report...
I20260926 20:46:05.320751 13635 tablet_bootstrap.cc:654] T 55a0d14968c8428bb9326574720ea59d P 95ac889929fb4ebea6bd466bfdf4faed: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:05.321069 13635 log.cc:824] T 55a0d14968c8428bb9326574720ea59d P 95ac889929fb4ebea6bd466bfdf4faed: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:05.324857 13635 tablet_bootstrap.cc:492] T 55a0d14968c8428bb9326574720ea59d P 95ac889929fb4ebea6bd466bfdf4faed: No bootstrap required, opened a new log
I20260926 20:46:05.324945 13635 ts_tablet_manager.cc:1397] T 55a0d14968c8428bb9326574720ea59d P 95ac889929fb4ebea6bd466bfdf4faed: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 20:46:05.327574 13635 raft_consensus.cc:348] T 55a0d14968c8428bb9326574720ea59d P 95ac889929fb4ebea6bd466bfdf4faed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b459f2625914997aeefdaa46dab79b7" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45149 } } peers { permanent_uuid: "95ac889929fb4ebea6bd466bfdf4faed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41351 } } peers { permanent_uuid: "ca85a2b94e1f463584678e8934fbbee4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44043 } }
I20260926 20:46:05.327764 13635 raft_consensus.cc:374] T 55a0d14968c8428bb9326574720ea59d P 95ac889929fb4ebea6bd466bfdf4faed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:05.327801 13635 raft_consensus.cc:729] T 55a0d14968c8428bb9326574720ea59d P 95ac889929fb4ebea6bd466bfdf4faed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95ac889929fb4ebea6bd466bfdf4faed, State: Initialized, Role: FOLLOWER
I20260926 20:46:05.327915 13635 consensus_queue.cc:260] T 55a0d14968c8428bb9326574720ea59d P 95ac889929fb4ebea6bd466bfdf4faed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b459f2625914997aeefdaa46dab79b7" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45149 } } peers { permanent_uuid: "95ac889929fb4ebea6bd466bfdf4faed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41351 } } peers { permanent_uuid: "ca85a2b94e1f463584678e8934fbbee4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44043 } }
I20260926 20:46:05.328181 13635 ts_tablet_manager.cc:1428] T 55a0d14968c8428bb9326574720ea59d P 95ac889929fb4ebea6bd466bfdf4faed: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 20:46:05.328205 13466 heartbeater.cc:503] Master 127.0.98.126:40093 was elected leader, sending a full tablet report...
I20260926 20:46:05.377801 13639 raft_consensus.cc:482] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:46:05.377954 13639 raft_consensus.cc:504] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b459f2625914997aeefdaa46dab79b7" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45149 } } peers { permanent_uuid: "95ac889929fb4ebea6bd466bfdf4faed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41351 } } peers { permanent_uuid: "ca85a2b94e1f463584678e8934fbbee4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44043 } }
I20260926 20:46:05.378347 13639 leader_election.cc:288] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95ac889929fb4ebea6bd466bfdf4faed (127.0.98.66:41351), ca85a2b94e1f463584678e8934fbbee4 (127.0.98.65:44043)
I20260926 20:46:05.382455 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:05.378727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:34561 (local address 127.0.98.65:44043)
0926 20:46:05.378802 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:05.378805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:05.379007 (+ 202us) server_negotiation.cc:407] Connection header received
0926 20:46:05.379104 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:05.379105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:05.379118 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:05.379145 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:05.379368 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.380083 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.381497 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.382052 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.382099 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:05.382108 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:05.382332 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 20:46:05.382382 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:46:05.382467 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:05.378751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:39127 (local address 127.0.98.66:41351)
0926 20:46:05.378834 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:05.378836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:05.378877 (+ 41us) server_negotiation.cc:407] Connection header received
0926 20:46:05.378977 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:05.378978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:05.378991 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:05.379020 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:05.379250 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.380000 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.381472 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.382037 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.382085 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:05.382096 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:05.382360 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 20:46:05.382402 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:46:05.382719 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:05.378513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:41351 (local address 127.0.98.67:39127)
0926 20:46:05.378805 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:05.378817 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:05.378875 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:05.379094 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:05.379095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:05.379097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:05.379175 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:05.379177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.380103 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.380103 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:05.381364 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:05.381367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.382197 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.382198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:05.382274 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:05.382278 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:46:05.382582 (+ 304us) client_negotiation.cc:240] Negotiation successful
0926 20:46:05.382624 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":15,"thread_start_us":109,"threads_started":1}
I20260926 20:46:05.383026 13420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55a0d14968c8428bb9326574720ea59d" candidate_uuid: "5b459f2625914997aeefdaa46dab79b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95ac889929fb4ebea6bd466bfdf4faed" is_pre_election: true
I20260926 20:46:05.383087 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:05.378544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:44043 (local address 127.0.98.67:34561)
0926 20:46:05.378945 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:05.378955 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:05.379032 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:05.379211 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:05.379212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:05.379213 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:05.379280 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:05.379281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.380182 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.380183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:05.381394 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:05.381397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.382187 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.382189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:05.382258 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:05.382261 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:46:05.382953 (+ 692us) client_negotiation.cc:240] Negotiation successful
0926 20:46:05.382998 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":3,"thread_start_us":88,"threads_started":1}
I20260926 20:46:05.383186 13420 raft_consensus.cc:2454] T 55a0d14968c8428bb9326574720ea59d P 95ac889929fb4ebea6bd466bfdf4faed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b459f2625914997aeefdaa46dab79b7 in term 0.
I20260926 20:46:05.383432 13490 leader_election.cc:302] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b459f2625914997aeefdaa46dab79b7, 95ac889929fb4ebea6bd466bfdf4faed; no voters:
I20260926 20:46:05.383580 13639 raft_consensus.cc:2781] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:46:05.383580 13268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55a0d14968c8428bb9326574720ea59d" candidate_uuid: "5b459f2625914997aeefdaa46dab79b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca85a2b94e1f463584678e8934fbbee4" is_pre_election: true
I20260926 20:46:05.383637 13639 raft_consensus.cc:482] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:46:05.383661 13639 raft_consensus.cc:3037] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:05.383733 13268 raft_consensus.cc:2454] T 55a0d14968c8428bb9326574720ea59d P ca85a2b94e1f463584678e8934fbbee4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b459f2625914997aeefdaa46dab79b7 in term 0.
I20260926 20:46:05.384617 13639 raft_consensus.cc:504] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b459f2625914997aeefdaa46dab79b7" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45149 } } peers { permanent_uuid: "95ac889929fb4ebea6bd466bfdf4faed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41351 } } peers { permanent_uuid: "ca85a2b94e1f463584678e8934fbbee4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44043 } }
I20260926 20:46:05.384754 13639 leader_election.cc:288] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [CANDIDATE]: Term 1 election: Requested vote from peers 95ac889929fb4ebea6bd466bfdf4faed (127.0.98.66:41351), ca85a2b94e1f463584678e8934fbbee4 (127.0.98.65:44043)
I20260926 20:46:05.384940 13268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55a0d14968c8428bb9326574720ea59d" candidate_uuid: "5b459f2625914997aeefdaa46dab79b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca85a2b94e1f463584678e8934fbbee4"
I20260926 20:46:05.384980 13420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55a0d14968c8428bb9326574720ea59d" candidate_uuid: "5b459f2625914997aeefdaa46dab79b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95ac889929fb4ebea6bd466bfdf4faed"
I20260926 20:46:05.385012 13268 raft_consensus.cc:3037] T 55a0d14968c8428bb9326574720ea59d P ca85a2b94e1f463584678e8934fbbee4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:05.385052 13420 raft_consensus.cc:3037] T 55a0d14968c8428bb9326574720ea59d P 95ac889929fb4ebea6bd466bfdf4faed [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:05.386029 13268 raft_consensus.cc:2454] T 55a0d14968c8428bb9326574720ea59d P ca85a2b94e1f463584678e8934fbbee4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b459f2625914997aeefdaa46dab79b7 in term 1.
I20260926 20:46:05.386044 13420 raft_consensus.cc:2454] T 55a0d14968c8428bb9326574720ea59d P 95ac889929fb4ebea6bd466bfdf4faed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b459f2625914997aeefdaa46dab79b7 in term 1.
I20260926 20:46:05.386204 13488 leader_election.cc:302] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b459f2625914997aeefdaa46dab79b7, ca85a2b94e1f463584678e8934fbbee4; no voters:
I20260926 20:46:05.386344 13639 raft_consensus.cc:2781] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:46:05.386502 13639 raft_consensus.cc:686] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [term 1 LEADER]: Becoming Leader. State: Replica: 5b459f2625914997aeefdaa46dab79b7, State: Running, Role: LEADER
I20260926 20:46:05.386592 13639 consensus_queue.cc:237] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b459f2625914997aeefdaa46dab79b7" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45149 } } peers { permanent_uuid: "95ac889929fb4ebea6bd466bfdf4faed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41351 } } peers { permanent_uuid: "ca85a2b94e1f463584678e8934fbbee4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44043 } }
I20260926 20:46:05.387351 13134 catalog_manager.cc:5495] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b459f2625914997aeefdaa46dab79b7 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "5b459f2625914997aeefdaa46dab79b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b459f2625914997aeefdaa46dab79b7" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95ac889929fb4ebea6bd466bfdf4faed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca85a2b94e1f463584678e8934fbbee4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44043 } health_report { overall_health: UNKNOWN } } }
W20260926 20:46:05.397871 13333 tablet.cc:2369] T 55a0d14968c8428bb9326574720ea59d P ca85a2b94e1f463584678e8934fbbee4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:46:05.400377 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:05.397006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.0.98.67:45149)
0926 20:46:05.397047 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:05.397049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:05.397059 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:46:05.397126 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:05.397126 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:05.397147 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:05.397223 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:46:05.397413 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.398128 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.399222 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:05.399530 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:05.399597 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:05.399844 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:46:05.399970 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:46:05.400043 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:05.400268 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 20:46:05.400290 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:46:05.402096 13268 raft_consensus.cc:1261] T 55a0d14968c8428bb9326574720ea59d P ca85a2b94e1f463584678e8934fbbee4 [term 1 FOLLOWER]: Refusing update from remote peer 5b459f2625914997aeefdaa46dab79b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:46:05.402541 13639 consensus_queue.cc:1035] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca85a2b94e1f463584678e8934fbbee4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:46:05.402698 13420 raft_consensus.cc:1261] T 55a0d14968c8428bb9326574720ea59d P 95ac889929fb4ebea6bd466bfdf4faed [term 1 FOLLOWER]: Refusing update from remote peer 5b459f2625914997aeefdaa46dab79b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:46:05.402999 13639 consensus_queue.cc:1035] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95ac889929fb4ebea6bd466bfdf4faed" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:46:05.410820 13134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:46:05.410918 13134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:46:05.411568 13134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57879009e1c6483fb5ed27868f10bd77: Sending DeleteTablet for 3 replicas of tablet 55a0d14968c8428bb9326574720ea59d
I20260926 20:46:05.412043 13534 tablet_service.cc:1514] Processing DeleteTablet for tablet 55a0d14968c8428bb9326574720ea59d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:46:05 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:46134
I20260926 20:46:05.412250 13400 tablet_service.cc:1514] Processing DeleteTablet for tablet 55a0d14968c8428bb9326574720ea59d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:46:05 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:40132
I20260926 20:46:05.412572 13254 tablet_service.cc:1514] Processing DeleteTablet for tablet 55a0d14968c8428bb9326574720ea59d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:46:05 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:32940
I20260926 20:46:05.413105 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 13172
I20260926 20:46:05.414466 13648 mvcc.cc:205] Tried to move back new op lower bound from 7333705995884490752 to 7333705995823984640. Current Snapshot: MvccSnapshot[applied={T|T < 7333705995884490752}]
I20260926 20:46:05.415853 13656 tablet_replica.cc:330] T 55a0d14968c8428bb9326574720ea59d P 95ac889929fb4ebea6bd466bfdf4faed: stopping tablet replica
I20260926 20:46:05.416030 13656 raft_consensus.cc:2229] T 55a0d14968c8428bb9326574720ea59d P 95ac889929fb4ebea6bd466bfdf4faed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:46:05.416163 13656 raft_consensus.cc:2258] T 55a0d14968c8428bb9326574720ea59d P 95ac889929fb4ebea6bd466bfdf4faed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:46:05.417418 13655 tablet_replica.cc:330] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7: stopping tablet replica
I20260926 20:46:05.417510 13655 raft_consensus.cc:2229] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:46:05.417591 13655 raft_consensus.cc:2258] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:46:05.418088 13655 ts_tablet_manager.cc:1905] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:46:05.420150 13655 ts_tablet_manager.cc:1918] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:46:05.420662 13656 ts_tablet_manager.cc:1905] T 55a0d14968c8428bb9326574720ea59d P 95ac889929fb4ebea6bd466bfdf4faed: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:46:05.421599 13119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:44043 (error 108)
I20260926 20:46:05.422134 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 13336
I20260926 20:46:05.424427 13655 log.cc:1196] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/55a0d14968c8428bb9326574720ea59d
I20260926 20:46:05.424952 13655 ts_tablet_manager.cc:1939] T 55a0d14968c8428bb9326574720ea59d P 5b459f2625914997aeefdaa46dab79b7: Deleting consensus metadata
I20260926 20:46:05.425387 13119 catalog_manager.cc:4841] TS 5b459f2625914997aeefdaa46dab79b7 (127.0.98.67:45149): tablet 55a0d14968c8428bb9326574720ea59d (table test-table [id=cd388c6fd0db4f47852d513f4c4e2a43]) successfully deleted
I20260926 20:46:05.425819 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:05.425654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:44043 (local address 127.0.0.1:32948)
0926 20:46:05.425725 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:05.425777 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:44043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 20:46:05.425930 13119 catalog_manager.cc:4584] TS ca85a2b94e1f463584678e8934fbbee4 (127.0.98.65:44043): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55a0d14968c8428bb9326574720ea59d: Network error: Client connection negotiation failed: client connection to 127.0.98.65:44043: connect: Connection refused (error 111)
I20260926 20:46:05.429291 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 13469
I20260926 20:46:05.430884 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:05.430764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:41351 (local address 127.0.0.1:40138)
0926 20:46:05.430811 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:05.430845 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.66:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 20:46:05.435098 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 13101
I20260926 20:46:05.441249 393 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 20:46:05.785673 393 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 20:46:05.785817 393 mini_ranger.cc:66] Stopping Ranger...
I20260926 20:46:06.130223 393 mini_ranger.cc:68] Stopped Ranger
2026-09-26 20:46:06.130 UTC [12168] LOG: received smart shutdown request
2026-09-26 20:46:06.134 UTC [12168] LOG: background worker "logical replication launcher" (PID 12177) exited with exit code 1
2026-09-26 20:46:06.134 UTC [12172] LOG: shutting down
2026-09-26 20:46:06.309 UTC [12168] LOG: database system is shut down
2026-09-26T20:46:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47472 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:46:06 dist-test-slave-4hd3 krb5kdc[13669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:46:06 dist-test-slave-4hd3 krb5kdc[13669](info): set up 2 sockets
Sep 26 20:46:06 dist-test-slave-4hd3 krb5kdc[13669](info): commencing operation
krb5kdc: starting...
W20260926 20:46:07.459669 393 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:46:07 dist-test-slave-4hd3 krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:46:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:46:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:46:07.575011 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:45199
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:45199,127.0.98.125:45553,127.0.98.124:35749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:46:07.655026 13682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:07.655256 13682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:07.655292 13682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:07.656415 13682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:46:07.656484 13682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:07.656508 13682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:07.656529 13682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:46:07.656558 13682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:46:07.658005 13682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:45199,127.0.98.125:45553,127.0.98.124:35749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:45199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:07.658278 13682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:07.658499 13682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:07.663305 13688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:07.663295 13687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:07.663538 13682 server_base.cc:1034] running on GCE node
W20260926 20:46:07.663295 13690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:07.663873 13682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:07.664165 13682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:07.665333 13682 hybrid_clock.cc:648] HybridClock initialized: now 1790455567665337 us; error 48 us; skew 500 ppm
Sep 26 20:46:07 dist-test-slave-4hd3 krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:07.673081 13682 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:46:07.673519 13682 webserver.cc:460] Webserver started at http://127.0.98.126:33895/ using document root <none> and password file <none>
I20260926 20:46:07.673707 13682 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:07.673748 13682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:07.673839 13682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:46:07.674856 13682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "d6d36a872225447a8e0a33f651240511"
format_stamp: "Formatted at 2026-09-26 20:46:07 on dist-test-slave-4hd3"
server_key: "338790bb65a652f8b0593e6927f189f9"
server_key_iv: "072b1e3c007d9861e096e4d29680eea5"
server_key_version: "encryptionkey@0"
I20260926 20:46:07.675153 13682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "d6d36a872225447a8e0a33f651240511"
format_stamp: "Formatted at 2026-09-26 20:46:07 on dist-test-slave-4hd3"
server_key: "338790bb65a652f8b0593e6927f189f9"
server_key_iv: "072b1e3c007d9861e096e4d29680eea5"
server_key_version: "encryptionkey@0"
I20260926 20:46:07.676528 13682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 20:46:07.677227 13697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:07.677464 13682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:46:07.677539 13682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "d6d36a872225447a8e0a33f651240511"
format_stamp: "Formatted at 2026-09-26 20:46:07 on dist-test-slave-4hd3"
server_key: "338790bb65a652f8b0593e6927f189f9"
server_key_iv: "072b1e3c007d9861e096e4d29680eea5"
server_key_version: "encryptionkey@0"
I20260926 20:46:07.677630 13682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:07.693455 13682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:07.695369 13682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:07.695528 13682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:07.699208 13682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:45199
I20260926 20:46:07.699262 13749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:45199 every 8 connection(s)
I20260926 20:46:07.699600 13682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:46:07.700546 13750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:07.702091 13750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:46:07 dist-test-slave-4hd3 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:07.709652 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 13682
I20260926 20:46:07.709776 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:46:07.709975 393 external_mini_cluster.cc:1428] Setting key 19adba914f8c78d29a7314430ddba3d3
I20260926 20:46:07.711964 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.702772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:45199 (local address 127.0.0.1:49270)
0926 20:46:07.702928 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.702965 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:07.703410 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:07.703698 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:07.703700 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:07.703951 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:07.704100 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:07.704104 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.705217 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.705218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:07.705936 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:07.705938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.706463 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.706464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:07.706528 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:07.707001 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:07.707009 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:07.708468 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:07.710364 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:07.710371 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:07.710373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:07.710645 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:07.710887 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:07.710888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:07.710888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:07.710999 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:07.711469 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:07.711475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:07.711528 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 20:46:07.711605 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:46:07.711778 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260926 20:46:07.712045 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.702818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49270 (local address 127.0.98.126:45199)
0926 20:46:07.702967 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:07.702980 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:07.703045 (+ 65us) server_negotiation.cc:407] Connection header received
0926 20:46:07.703560 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:07.703562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:07.703590 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:07.703674 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:07.704213 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.705063 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.706017 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.706362 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.706445 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:07.708559 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:07.708566 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:07.708566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:07.708572 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:07.710279 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:07.710721 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:07.710723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:07.710723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:07.710816 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:07.711062 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:07.711063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:07.711063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:07.711381 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:07.711472 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:07.711659 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 20:46:07.711861 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260926 20:46:07.712818 13750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:46:07.713223 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.713098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:45553 (local address 127.0.0.1:36380)
0926 20:46:07.713149 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.713198 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 20:46:07.713385 13699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.125:45553: connect: Connection refused (error 111)
I20260926 20:46:07.713807 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.713731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:45553 (local address 127.0.0.1:36396)
0926 20:46:07.713759 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.713783 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 20:46:07.713912 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:45553: Network error: Client connection negotiation failed: client connection to 127.0.98.125:45553: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.125.keytab.
Entry for principal kudu/127.0.98.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.125.keytab.
I20260926 20:46:07.732856 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:45553
--webserver_interface=127.0.98.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.125.keytab
--principal=kudu/127.0.98.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:45199,127.0.98.125:45553,127.0.98.124:35749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:46:07.763142 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } attempt: 1
I20260926 20:46:07.763621 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.763482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:45553 (local address 127.0.0.1:36412)
0926 20:46:07.763553 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.763592 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 20:46:07.763996 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.763901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:45553 (local address 127.0.0.1:36424)
0926 20:46:07.763947 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.763974 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 20:46:07.764077 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:45553: Network error: Client connection negotiation failed: client connection to 127.0.98.125:45553: connect: Connection refused (error 111)
W20260926 20:46:07.813385 13757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:07.813602 13757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:07.813639 13757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:07.814715 13757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:46:07.814777 13757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:07.814803 13757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:07.814834 13757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:46:07.814858 13757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:46:07.816300 13757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:45199,127.0.98.125:45553,127.0.98.124:35749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.125.keytab
--principal=kudu/127.0.98.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:45553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:07.816557 13757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:07.816759 13757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:07.821485 13763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:07.821542 13762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:07.821506 13765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:07.821802 13757 server_base.cc:1034] running on GCE node
I20260926 20:46:07.822064 13757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:07.822342 13757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:07.823503 13757 hybrid_clock.cc:648] HybridClock initialized: now 1790455567823497 us; error 33 us; skew 500 ppm
Sep 26 20:46:07 dist-test-slave-4hd3 krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:07.825902 13757 init.cc:377] Logged in from keytab as kudu/127.0.98.125@KRBTEST.COM (short username kudu)
I20260926 20:46:07.826375 13757 webserver.cc:460] Webserver started at http://127.0.98.125:37195/ using document root <none> and password file <none>
I20260926 20:46:07.826606 13757 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:07.826673 13757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:07.826803 13757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:46:07.827978 13757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-1/data/instance:
uuid: "1d2acf658aa44efbb8e6220ed142269d"
format_stamp: "Formatted at 2026-09-26 20:46:07 on dist-test-slave-4hd3"
server_key: "03245e9f20b66b4e0fa219f12d6c791e"
server_key_iv: "65f9336742b4e4916dd58a313ff07e13"
server_key_version: "encryptionkey@0"
I20260926 20:46:07.828327 13757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-1/wal/instance:
uuid: "1d2acf658aa44efbb8e6220ed142269d"
format_stamp: "Formatted at 2026-09-26 20:46:07 on dist-test-slave-4hd3"
server_key: "03245e9f20b66b4e0fa219f12d6c791e"
server_key_iv: "65f9336742b4e4916dd58a313ff07e13"
server_key_version: "encryptionkey@0"
I20260926 20:46:07.829614 13757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:46:07.830343 13772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:07.830518 13757 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:46:07.830610 13757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-1/wal
uuid: "1d2acf658aa44efbb8e6220ed142269d"
format_stamp: "Formatted at 2026-09-26 20:46:07 on dist-test-slave-4hd3"
server_key: "03245e9f20b66b4e0fa219f12d6c791e"
server_key_iv: "65f9336742b4e4916dd58a313ff07e13"
server_key_version: "encryptionkey@0"
I20260926 20:46:07.830695 13757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:07.832208 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } attempt: 2
I20260926 20:46:07.832590 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.832466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:45553 (local address 127.0.0.1:36426)
0926 20:46:07.832530 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.832564 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 20:46:07.832916 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.832816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:45553 (local address 127.0.0.1:36436)
0926 20:46:07.832864 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.832889 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 20:46:07.832994 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:45553: Network error: Client connection negotiation failed: client connection to 127.0.98.125:45553: connect: Connection refused (error 111)
I20260926 20:46:07.839138 13757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:07.841012 13757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:07.841167 13757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:07.845244 13757 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.125:45553
I20260926 20:46:07.845280 13824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.125:45553 every 8 connection(s)
I20260926 20:46:07.845657 13757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-1/data/info.pb
I20260926 20:46:07.846460 13825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:07.847884 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 13757
I20260926 20:46:07.847968 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-1/wal/instance
I20260926 20:46:07.848163 393 external_mini_cluster.cc:1428] Setting key 290e74b50a9c4164258833db07465334
I20260926 20:46:07.848122 13825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:46:07 dist-test-slave-4hd3 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.124@KRBTEST.COM; defaulting to no policy
I20260926 20:46:07.857309 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.848910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.0.98.126:45199)
0926 20:46:07.848953 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:07.848955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:07.849191 (+ 236us) server_negotiation.cc:407] Connection header received
0926 20:46:07.849724 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:07.849725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:07.849739 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:07.849809 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:07.850390 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.850886 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.851987 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.852213 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.852298 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:07.854338 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:07.854339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:07.854339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:07.854347 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:07.855903 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:07.856316 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:07.856318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:07.856318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:07.856408 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:07.856656 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:07.856657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:07.856657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:07.856778 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:07.856848 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:07.857124 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 20:46:07.857210 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:46:07.857503 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.848891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:45199 (local address 127.0.0.1:49276)
0926 20:46:07.849075 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.849108 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:07.849615 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:07.849875 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:07.849879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:07.850112 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:07.850290 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:07.850303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.850980 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.850981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:07.851905 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:07.851907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.852314 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.852315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:07.852434 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:07.852955 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:07.852963 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:07.854236 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:07.856004 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:07.856011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:07.856013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:07.856228 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:07.856479 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:07.856480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:07.856480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:07.856582 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:07.856861 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:07.856868 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:07.856994 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 20:46:07.857094 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 20:46:07.857241 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.124@KRBTEST.COM" created.
I20260926 20:46:07.858086 13825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:46:07 dist-test-slave-4hd3 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260926 20:46:07.866500 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.859435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36440 (local address 127.0.98.125:45553)
0926 20:46:07.859589 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:07.859594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:07.859612 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:46:07.859634 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:07.859636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:07.859650 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:07.859780 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:07.860052 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.860745 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.861512 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.861810 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.861899 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:07.863912 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:07.863918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:07.863919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:07.863925 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:07.865391 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:07.865726 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:07.865728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:07.865728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:07.865774 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:07.865953 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:07.865954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:07.865954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:07.866185 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:07.866248 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:07.866360 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 20:46:07.866426 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":101,"threads_started":1}
I20260926 20:46:07.866577 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.858357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:45553 (local address 127.0.0.1:36440)
0926 20:46:07.858410 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.858421 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:07.858455 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:07.859781 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:07.859782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:07.859925 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:07.859969 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:07.859970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.860845 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.860846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:07.861417 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:07.861419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.861898 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.861899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:07.861963 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:07.862390 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:07.862395 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:07.863834 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:07.865468 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:07.865469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:07.865470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:07.865655 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:07.865828 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:07.865829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:07.865829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:07.865882 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:07.866246 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:07.866247 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:07.866282 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 20:46:07.866336 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:46:07.866471 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":26}
I20260926 20:46:07.867225 13825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:46:07.867522 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.867437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35749 (local address 127.0.0.1:55302)
0926 20:46:07.867465 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.867504 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 20:46:07.867681 13773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35749: connect: Connection refused (error 111)
I20260926 20:46:07.868147 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.868064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35749 (local address 127.0.0.1:55308)
0926 20:46:07.868108 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.868132 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 20:46:07.868269 13825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:35749: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35749: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.124.keytab.
Entry for principal kudu/127.0.98.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.124.keytab.
I20260926 20:46:07.872262 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:35749
--webserver_interface=127.0.98.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.124.keytab
--principal=kudu/127.0.98.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:45199,127.0.98.125:45553,127.0.98.124:35749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:46:07.917559 13825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } attempt: 1
I20260926 20:46:07.918059 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.917923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35749 (local address 127.0.0.1:55314)
0926 20:46:07.917988 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.918029 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 20:46:07.918406 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.918309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35749 (local address 127.0.0.1:55320)
0926 20:46:07.918354 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.918381 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 20:46:07.918499 13825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:35749: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35749: connect: Connection refused (error 111)
I20260926 20:46:07.924127 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } attempt: 3
Sep 26 20:46:07 dist-test-slave-4hd3 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260926 20:46:07.931407 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.924536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36448 (local address 127.0.98.125:45553)
0926 20:46:07.924596 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:07.924597 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:07.924608 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:46:07.924670 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:07.924671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:07.924682 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:07.924751 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:07.925081 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.925633 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.926299 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.926511 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.926584 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:07.928775 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:07.928776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:07.928777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:07.928785 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:07.930213 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:07.930613 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:07.930614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:07.930614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:07.930663 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:07.930887 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:07.930888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:07.930889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:07.931008 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:07.931074 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:07.931243 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 20:46:07.931312 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:46:07.931417 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.924474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:45553 (local address 127.0.0.1:36448)
0926 20:46:07.924533 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.924542 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:07.924602 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:07.924759 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:07.924760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:07.924925 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:07.924988 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:07.924990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.925710 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.925712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:07.926208 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:07.926209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.926581 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.926582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:07.926664 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:07.927168 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:07.927172 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:07.928665 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:07.930301 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:07.930303 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:07.930303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:07.930540 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:07.930726 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:07.930727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:07.930727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:07.930795 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:07.931085 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:07.931086 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:07.931135 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 20:46:07.931213 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:46:07.931319 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 20:46:07.932086 13750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:46:07.932435 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.932324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35749 (local address 127.0.0.1:55324)
0926 20:46:07.932380 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.932411 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 20:46:07.932765 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.932699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35749 (local address 127.0.0.1:55340)
0926 20:46:07.932731 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.932751 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260926 20:46:07.932862 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:35749: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35749: connect: Connection refused (error 111)
W20260926 20:46:07.954272 13832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:07.954488 13832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:07.954510 13832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:07.955665 13832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:46:07.955752 13832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:07.955786 13832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:07.955834 13832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:46:07.955852 13832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:46:07.957361 13832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:45199,127.0.98.125:45553,127.0.98.124:35749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.124.keytab
--principal=kudu/127.0.98.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:35749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:07.957609 13832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:07.957861 13832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:07.962736 13838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:07.962745 13837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:07.962745 13840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:07.963037 13832 server_base.cc:1034] running on GCE node
I20260926 20:46:07.963302 13832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:07.963599 13832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:07.963994 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } attempt: 1
I20260926 20:46:07.964353 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.964238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35749 (local address 127.0.0.1:55356)
0926 20:46:07.964295 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.964328 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 20:46:07.964622 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.964549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35749 (local address 127.0.0.1:55368)
0926 20:46:07.964585 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.964607 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 20:46:07.964715 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:35749: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35749: connect: Connection refused (error 111)
I20260926 20:46:07.964771 13832 hybrid_clock.cc:648] HybridClock initialized: now 1790455567964745 us; error 38 us; skew 500 ppm
Sep 26 20:46:07 dist-test-slave-4hd3 krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:07.967376 13832 init.cc:377] Logged in from keytab as kudu/127.0.98.124@KRBTEST.COM (short username kudu)
I20260926 20:46:07.967876 13832 webserver.cc:460] Webserver started at http://127.0.98.124:36657/ using document root <none> and password file <none>
I20260926 20:46:07.968101 13832 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:07.968168 13832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:07.968322 13832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:46:07.969393 13832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-2/data/instance:
uuid: "0d9dbeafa7224bdcba1f7d8ba2684698"
format_stamp: "Formatted at 2026-09-26 20:46:07 on dist-test-slave-4hd3"
server_key: "690151856ecd971a2d0e58e5c0dcc5b7"
server_key_iv: "fd60413329a5958923fa460b8261b60b"
server_key_version: "encryptionkey@0"
I20260926 20:46:07.969761 13832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-2/wal/instance:
uuid: "0d9dbeafa7224bdcba1f7d8ba2684698"
format_stamp: "Formatted at 2026-09-26 20:46:07 on dist-test-slave-4hd3"
server_key: "690151856ecd971a2d0e58e5c0dcc5b7"
server_key_iv: "fd60413329a5958923fa460b8261b60b"
server_key_version: "encryptionkey@0"
I20260926 20:46:07.971078 13832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:46:07.971901 13847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:07.972074 13832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:46:07.972170 13832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-2/wal
uuid: "0d9dbeafa7224bdcba1f7d8ba2684698"
format_stamp: "Formatted at 2026-09-26 20:46:07 on dist-test-slave-4hd3"
server_key: "690151856ecd971a2d0e58e5c0dcc5b7"
server_key_iv: "fd60413329a5958923fa460b8261b60b"
server_key_version: "encryptionkey@0"
I20260926 20:46:07.972252 13832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:07.986632 13825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } attempt: 2
I20260926 20:46:07.987011 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.986889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35749 (local address 127.0.0.1:55384)
0926 20:46:07.986948 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.986984 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 20:46:07.987026 13832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:07.987408 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.987305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35749 (local address 127.0.0.1:55396)
0926 20:46:07.987355 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.987385 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 20:46:07.987542 13825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:35749: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35749: connect: Connection refused (error 111)
I20260926 20:46:07.989187 13832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:07.989343 13832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:07.993129 13832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.124:35749
I20260926 20:46:07.993196 13899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.124:35749 every 8 connection(s)
I20260926 20:46:07.993516 13832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-2/data/info.pb
I20260926 20:46:07.994382 13900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:07.995873 13900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:46:07.996551 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 13832
I20260926 20:46:07.996665 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455416624047-393-0/minicluster-data/master-2/wal/instance
I20260926 20:46:07.996869 393 external_mini_cluster.cc:1428] Setting key 432b7baf44e7bd30072472cfeaf6ef9d
Sep 26 20:46:07 dist-test-slave-4hd3 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
Sep 26 20:46:07 dist-test-slave-4hd3 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260926 20:46:08.005475 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.996537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49284 (local address 127.0.98.126:45199)
0926 20:46:07.996606 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:07.996608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:07.996866 (+ 258us) server_negotiation.cc:407] Connection header received
0926 20:46:07.997471 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:07.997473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:07.997485 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:07.997546 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:07.998084 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.998565 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.999536 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.999781 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.999860 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:08.002120 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:08.002121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:08.002122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:08.002130 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:08.003920 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.004372 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.004373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.004374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.004432 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.004719 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.004720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.004720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.004870 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:08.004942 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:08.005244 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 20:46:08.005373 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:46:08.005614 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.996490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:45199 (local address 127.0.0.1:49284)
0926 20:46:07.996741 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:07.996783 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:07.997373 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:07.997573 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:07.997578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:07.997826 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:07.997992 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:07.997996 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.998650 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.998651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:07.999436 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:07.999438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.999874 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.999875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:07.999998 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:08.000462 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:08.000470 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:08.002032 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:08.004001 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:08.004006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:08.004008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:08.004295 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:08.004502 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:08.004502 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:08.004503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:08.004649 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:08.004945 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:08.004952 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:08.005117 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 20:46:08.005198 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 20:46:08.005343 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":71,"thread_start_us":71,"threads_started":1}
I20260926 20:46:08.005952 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:07.997236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55400 (local address 127.0.98.124:35749)
0926 20:46:07.997445 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:07.997450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:07.997487 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:46:07.997570 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:07.997572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:07.997587 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:07.997679 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:07.998285 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:07.999071 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:07.999812 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.000112 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.000198 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:08.002645 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:08.002652 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:08.002653 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:08.002659 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:08.004420 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.004798 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.004800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.004801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.004870 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.005076 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.005076 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.005077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.005226 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:08.005381 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:08.005454 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 20:46:08.005529 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":151,"threads_started":1}
I20260926 20:46:08.006206 13900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:46:08 dist-test-slave-4hd3 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
Sep 26 20:46:08 dist-test-slave-4hd3 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:08.013119 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:08.006503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36464 (local address 127.0.98.125:45553)
0926 20:46:08.006549 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:08.006553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:08.006567 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:46:08.006688 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:08.006690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:08.006702 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:08.006757 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:08.007050 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.007651 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.008394 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.008679 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.008727 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:08.010466 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:08.010467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:08.010468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:08.010473 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:08.012022 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.012401 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.012402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.012402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.012447 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.012637 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.012638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.012638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.012776 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:08.012835 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:08.013007 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 20:46:08.013061 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:46:08.013149 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:08.006441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:45553 (local address 127.0.0.1:36464)
0926 20:46:08.006478 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:08.006488 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:08.006582 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:08.006761 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:08.006762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:08.006936 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:08.006977 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:08.006978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.007740 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.007741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:08.008298 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:08.008299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.008745 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.008746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:08.008811 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:08.009211 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:08.009216 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:08.010387 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:08.012098 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:08.012099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:08.012100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:08.012331 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:08.012518 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:08.012518 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:08.012519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:08.012577 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:08.012843 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:08.012844 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:08.012896 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 20:46:08.012991 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 20:46:08.013069 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 20:46:08.013485 13900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:46:08.014667 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:08.006674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.0.98.126:45199)
0926 20:46:08.006726 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:08.006727 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:08.006735 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:46:08.006793 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:08.006794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:08.006803 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:08.006851 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:08.007377 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.007880 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.008591 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.008813 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.008884 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:08.011124 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:08.011125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:08.011125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:08.011132 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:08.013489 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.013901 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.013902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.013903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.013961 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.014157 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.014158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.014158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.014266 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:08.014463 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:08.014499 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 20:46:08.014574 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 20:46:08 dist-test-slave-4hd3 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
Sep 26 20:46:08 dist-test-slave-4hd3 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260926 20:46:08.020758 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:08.013712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35749 (local address 127.0.0.1:55414)
0926 20:46:08.013762 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:08.013772 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:08.013823 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:08.013946 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:08.013946 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:08.014066 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:08.014089 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:08.014090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.014711 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.014713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:08.015324 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:08.015326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.015873 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.015874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:08.015920 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:08.016745 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:08.016749 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:08.017976 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:08.019697 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:08.019698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:08.019698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:08.019944 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:08.020150 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:08.020151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:08.020152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:08.020207 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:08.020457 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:08.020458 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:08.020494 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 20:46:08.020562 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 20:46:08.020652 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:46:08.020820 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:08.013772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55414 (local address 127.0.98.124:35749)
0926 20:46:08.013802 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:08.013804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:08.013811 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:46:08.013879 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:08.013879 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:08.013889 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:08.013942 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:08.014157 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.014628 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.015474 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.015819 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.015876 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:08.018065 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:08.018066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:08.018066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:08.018072 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:08.019567 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.020025 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.020026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.020026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.020073 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.020273 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.020274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.020274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.020389 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:08.020464 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:08.020589 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 20:46:08.020703 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":16}
I20260926 20:46:08.022735 13900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698: Bootstrap starting.
I20260926 20:46:08.023033 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:08.015584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36478 (local address 127.0.98.125:45553)
0926 20:46:08.015632 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:08.015634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:08.015640 (+ 6us) server_negotiation.cc:407] Connection header received
0926 20:46:08.015657 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:08.015658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:08.015667 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:08.015769 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:08.016199 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.016651 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.017340 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.017509 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.017558 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:08.020092 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:08.020093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:08.020093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:08.020100 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:08.021847 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.022153 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.022154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.022155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.022219 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.022470 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.022471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.022471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.022590 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:08.022758 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:08.022884 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 20:46:08.022957 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 20:46:08.023458 13900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:08.023680 13900 log.cc:824] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:08.024413 13900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698: No bootstrap required, opened a new log
I20260926 20:46:08.025287 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:46:08.026121 13900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } } peers { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } } peers { permanent_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } }
I20260926 20:46:08.026309 13900 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:08.026369 13900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9dbeafa7224bdcba1f7d8ba2684698, State: Initialized, Role: FOLLOWER
I20260926 20:46:08.026564 13900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } } peers { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } } peers { permanent_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } }
I20260926 20:46:08.026892 13900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:46:08.026852 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } } peers { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } } peers { permanent_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } } }
I20260926 20:46:08.027139 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:46:08.031406 13924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:46:08.031507 13924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:46:08.029088 13832 master_runner.cc:186] Error getting master registration for 127.0.98.126:45199: OK, instance_id { permanent_uuid: "d6d36a872225447a8e0a33f651240511" instance_seqno: 1790455567698234 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:46:08.032725 13832 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:46:08.035799 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:08.027010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.0.98.126:45199)
0926 20:46:08.027086 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:08.027088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:08.027095 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:46:08.027124 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:08.027125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:08.027135 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:08.027185 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:08.027933 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.028655 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.029655 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.029944 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.029983 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:08.031464 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:08.031466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:08.031466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:08.031474 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:08.033201 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.034674 (+ 1473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.034676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.034676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.034737 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.034945 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.034946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.034947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.035082 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:08.035178 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:08.035580 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 20:46:08.035662 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:46:08.036302 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:08.026917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55430 (local address 127.0.98.124:35749)
0926 20:46:08.027056 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:08.027058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:08.027066 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:46:08.027084 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:08.027086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:08.027096 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:08.027157 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:08.027961 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.028656 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.029792 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.030070 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.030105 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:08.033142 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:08.033144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:08.033144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:08.033152 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:08.034909 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.035254 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.035255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.035255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.035310 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.035551 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.035552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.035552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.035706 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:08.035818 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:08.036137 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 20:46:08.036214 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260926 20:46:08.036774 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:08.027008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.0.98.125:45553)
0926 20:46:08.027048 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:08.027050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:08.028058 (+ 1008us) server_negotiation.cc:407] Connection header received
0926 20:46:08.028160 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:08.028161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:08.028172 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:08.028204 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:08.029086 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.029731 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.030698 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.031045 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.031121 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:08.033159 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:08.033161 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:08.033161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:08.033168 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:08.035066 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.035537 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.035539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.035539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.035596 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.035849 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.035850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.035850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.035974 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:08.036042 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:08.036585 (+ 543us) server_negotiation.cc:299] Negotiation successful
0926 20:46:08.036682 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:46:08.039829 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } attempt: 2
Sep 26 20:46:08 dist-test-slave-4hd3 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260926 20:46:08.046581 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:08.040151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55446 (local address 127.0.98.124:35749)
0926 20:46:08.040190 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:08.040193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:08.040200 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:46:08.040223 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:08.040224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:08.040233 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:08.040287 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:08.040554 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.041047 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.041719 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.041991 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.042284 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:08.044086 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:08.044087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:08.044088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:08.044094 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:08.045482 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.045825 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.045826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.045826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.045882 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.046078 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.046079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.046079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.046191 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:08.046261 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:08.046426 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 20:46:08.046495 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:46:08.046581 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:08.040080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35749 (local address 127.0.0.1:55446)
0926 20:46:08.040129 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:08.040139 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:08.040178 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:08.040291 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:08.040292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:08.040442 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:08.040478 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:08.040479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.041123 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.041124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:08.041615 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:08.041617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.042078 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.042079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:08.042135 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:08.042584 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:08.042589 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:08.044003 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:08.045584 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:08.045585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:08.045585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:08.045754 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:08.045949 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:08.045950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:08.045950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:08.046007 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:08.046253 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:08.046254 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:08.046297 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 20:46:08.046383 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 20:46:08.046482 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 20:46:08.048481 13750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511: Bootstrap starting.
I20260926 20:46:08.049203 13750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:08.049520 13750 log.cc:824] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:08.050420 13750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511: No bootstrap required, opened a new log
I20260926 20:46:08.052493 13750 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } } peers { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } } peers { permanent_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } }
I20260926 20:46:08.052637 13750 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:08.052673 13750 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d36a872225447a8e0a33f651240511, State: Initialized, Role: FOLLOWER
I20260926 20:46:08.052788 13750 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } } peers { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } } peers { permanent_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } }
I20260926 20:46:08.053102 13750 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:46:08.053069 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } } peers { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } } peers { permanent_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } } }
I20260926 20:46:08.053179 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:46:08.055040 13938 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:46:08.055150 13938 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:46:08.055778 13682 master_runner.cc:186] Error getting master registration for 127.0.98.125:45553: OK, instance_id { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" instance_seqno: 1790455567844091 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:46:08.057997 13682 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:46:08.062350 13912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:46:08.062471 13912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } } peers { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } } peers { permanent_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } }
I20260926 20:46:08.062762 13912 leader_election.cc:288] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6d36a872225447a8e0a33f651240511 (127.0.98.126:45199), 1d2acf658aa44efbb8e6220ed142269d (127.0.98.125:45553)
I20260926 20:46:08.063133 13724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6d36a872225447a8e0a33f651240511" is_pre_election: true
I20260926 20:46:08.063290 13724 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d9dbeafa7224bdcba1f7d8ba2684698 in term 0.
I20260926 20:46:08.063874 13850 leader_election.cc:302] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9dbeafa7224bdcba1f7d8ba2684698, d6d36a872225447a8e0a33f651240511; no voters:
I20260926 20:46:08.064479 13912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:46:08.064548 13912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:46:08.064571 13912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:08.065479 13912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } } peers { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } } peers { permanent_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } }
I20260926 20:46:08.065610 13912 leader_election.cc:288] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [CANDIDATE]: Term 1 election: Requested vote from peers d6d36a872225447a8e0a33f651240511 (127.0.98.126:45199), 1d2acf658aa44efbb8e6220ed142269d (127.0.98.125:45553)
I20260926 20:46:08.065807 13724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6d36a872225447a8e0a33f651240511"
I20260926 20:46:08.065876 13724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:08.066807 13724 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d9dbeafa7224bdcba1f7d8ba2684698 in term 1.
I20260926 20:46:08.066964 13850 leader_election.cc:302] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9dbeafa7224bdcba1f7d8ba2684698, d6d36a872225447a8e0a33f651240511; no voters:
I20260926 20:46:08.067045 13912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:46:08.067209 13912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [term 1 LEADER]: Becoming Leader. State: Replica: 0d9dbeafa7224bdcba1f7d8ba2684698, State: Running, Role: LEADER
I20260926 20:46:08.067301 13912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } } peers { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } } peers { permanent_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } }
I20260926 20:46:08.067665 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d9dbeafa7224bdcba1f7d8ba2684698. Latest consensus state: current_term: 1 leader_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } } peers { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } } peers { permanent_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } } }
I20260926 20:46:08.067795 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [sys.catalog]: This master's current role is: LEADER
I20260926 20:46:08.067978 13942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:46:08.068174 13942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:46:08.069605 13724 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511 [term 1 FOLLOWER]: Refusing update from remote peer 0d9dbeafa7224bdcba1f7d8ba2684698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:46:08.069962 13940 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:46:08.071211 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d9dbeafa7224bdcba1f7d8ba2684698. Latest consensus state: current_term: 1 leader_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } } peers { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } } peers { permanent_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } } }
I20260926 20:46:08.071293 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:46:08.072957 13943 mvcc.cc:205] Tried to move back new op lower bound from 7333706006811078656 to 7333706006804312064. Current Snapshot: MvccSnapshot[applied={T|T < 7333706006811078656}]
I20260926 20:46:08.073684 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } } peers { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } } peers { permanent_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } } }
I20260926 20:46:08.073766 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [sys.catalog]: This master's current role is: LEADER
I20260926 20:46:08.074172 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } } peers { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } } peers { permanent_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } } }
I20260926 20:46:08.074265 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:46:08.074688 13942 catalog_manager.cc:1347] Generated new cluster ID: 8dfd1ec9efed47f7aeadb85d2994894a
I20260926 20:46:08.074735 13942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:46:08.078697 13825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } attempt: 3
Sep 26 20:46:08 dist-test-slave-4hd3 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260926 20:46:08.087354 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:08.078945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35749 (local address 127.0.0.1:55448)
0926 20:46:08.079024 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:08.079035 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:08.079073 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:08.079401 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:08.079402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:08.079588 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:08.079637 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:08.079638 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.080691 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.080693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:08.081517 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:08.081519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.082082 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.082083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:08.082155 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:08.082655 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:08.082660 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:08.084268 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:08.086125 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:08.086126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:08.086127 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:08.086352 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:08.086602 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:08.086603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:08.086603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:08.086672 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:08.086988 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:08.086989 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:08.087046 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 20:46:08.087106 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 20:46:08.087195 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 20:46:08.087860 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:08.079140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55448 (local address 127.0.98.124:35749)
0926 20:46:08.079188 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:08.079190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:08.079196 (+ 6us) server_negotiation.cc:407] Connection header received
0926 20:46:08.079217 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:08.079218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:08.079226 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:08.079285 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:08.079746 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.080542 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.081631 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:08.081941 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:08.082006 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:08.084394 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:08.084396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:08.084396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:08.084402 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:08.086017 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.086446 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.086446 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.086447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.086518 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:08.086751 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:08.086751 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:08.086752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:08.086885 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:08.086942 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:08.087685 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 20:46:08.087775 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:46:08.089951 13825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d: Bootstrap starting.
I20260926 20:46:08.090648 13825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:08.090969 13825 log.cc:824] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:08.092293 13825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d: No bootstrap required, opened a new log
I20260926 20:46:08.094413 13825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } } peers { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } } peers { permanent_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } }
I20260926 20:46:08.094551 13825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:08.094591 13825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d2acf658aa44efbb8e6220ed142269d, State: Initialized, Role: FOLLOWER
I20260926 20:46:08.094723 13825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } } peers { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } } peers { permanent_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } }
I20260926 20:46:08.095027 13825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:46:08.095551 13799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d2acf658aa44efbb8e6220ed142269d" is_pre_election: true
I20260926 20:46:08.096017 13798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d2acf658aa44efbb8e6220ed142269d"
I20260926 20:46:08.088681 13942 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:46:08.096271 13797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:08.097009 13942 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:46:08.097637 13797 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d [term 1 FOLLOWER]: Refusing update from remote peer 0d9dbeafa7224bdcba1f7d8ba2684698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:46:08.097954 13912 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:46:08.099340 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } } peers { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } } peers { permanent_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } } }
I20260926 20:46:08.099474 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:46:08.099567 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d9dbeafa7224bdcba1f7d8ba2684698. Latest consensus state: current_term: 1 leader_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } } peers { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } } peers { permanent_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } } }
I20260926 20:46:08.099661 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:46:08.099762 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } } peers { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } } peers { permanent_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } } }
I20260926 20:46:08.099865 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:46:08.100672 13965 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:46:08.100723 13965 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:46:08.102211 13962 mvcc.cc:205] Tried to move back new op lower bound from 7333706006885998592 to 7333706006804312064. Current Snapshot: MvccSnapshot[applied={T|T < 7333706006811078656}]
I20260926 20:46:08.110750 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d36a872225447a8e0a33f651240511" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45199 } } peers { permanent_uuid: "1d2acf658aa44efbb8e6220ed142269d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45553 } } peers { permanent_uuid: "0d9dbeafa7224bdcba1f7d8ba2684698" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35749 } } }
I20260926 20:46:08.110862 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698 [sys.catalog]: This master's current role is: LEADER
I20260926 20:46:08.113082 13942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d9dbeafa7224bdcba1f7d8ba2684698: Generated new TSK 0
I20260926 20:46:08.113262 13942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:46:09.055694 13938 catalog_manager.cc:1259] Loaded cluster ID: 8dfd1ec9efed47f7aeadb85d2994894a
I20260926 20:46:09.055812 13938 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511: loading cluster ID for follower catalog manager: success
I20260926 20:46:09.056737 13938 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511: acquiring CA information for follower catalog manager: success
I20260926 20:46:09.057071 13938 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d6d36a872225447a8e0a33f651240511: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:46:09.101136 13965 catalog_manager.cc:1259] Loaded cluster ID: 8dfd1ec9efed47f7aeadb85d2994894a
I20260926 20:46:09.101208 13965 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d: loading cluster ID for follower catalog manager: success
I20260926 20:46:09.102066 13965 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d: acquiring CA information for follower catalog manager: success
I20260926 20:46:09.102388 13965 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1d2acf658aa44efbb8e6220ed142269d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:46:09.170699 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 13682
I20260926 20:46:09.176124 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 13757
I20260926 20:46:09.180954 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 13832
2026-09-26T20:46:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2756 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:46:09 dist-test-slave-4hd3 krb5kdc[13976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:46:09 dist-test-slave-4hd3 krb5kdc[13976](info): set up 2 sockets
Sep 26 20:46:09 dist-test-slave-4hd3 krb5kdc[13976](info): commencing operation
krb5kdc: starting...
W20260926 20:46:10.203969 393 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:46:10 dist-test-slave-4hd3 krb5kdc[13976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:46:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:46:10.318002 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:44399
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:44399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:46:10.397497 13989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:10.397740 13989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:10.397774 13989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:10.398892 13989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:46:10.398962 13989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:10.398984 13989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:10.399004 13989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:46:10.399034 13989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:46:10.400604 13989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:44399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:44399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:10.400866 13989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:10.401084 13989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:10.405887 13994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:10.405881 13995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:10.406057 13997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:10.405970 13989 server_base.cc:1034] running on GCE node
I20260926 20:46:10.406417 13989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:10.406714 13989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:10.407888 13989 hybrid_clock.cc:648] HybridClock initialized: now 1790455570407878 us; error 34 us; skew 500 ppm
Sep 26 20:46:10 dist-test-slave-4hd3 krb5kdc[13976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:10.410382 13989 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:46:10.410857 13989 webserver.cc:460] Webserver started at http://127.0.98.126:38597/ using document root <none> and password file <none>
I20260926 20:46:10.411103 13989 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:10.411170 13989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:10.411325 13989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:46:10.412420 13989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "3e17cf7bdde24e15a361c42894205a39"
format_stamp: "Formatted at 2026-09-26 20:46:10 on dist-test-slave-4hd3"
server_key: "07c0921b0bdf4c267abb3f74e5fd2234"
server_key_iv: "b6f548db38b26910ec1eb37fbdd1338b"
server_key_version: "encryptionkey@0"
I20260926 20:46:10.412773 13989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "3e17cf7bdde24e15a361c42894205a39"
format_stamp: "Formatted at 2026-09-26 20:46:10 on dist-test-slave-4hd3"
server_key: "07c0921b0bdf4c267abb3f74e5fd2234"
server_key_iv: "b6f548db38b26910ec1eb37fbdd1338b"
server_key_version: "encryptionkey@0"
I20260926 20:46:10.414108 13989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:46:10.414827 14004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:10.415164 13989 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:46:10.415283 13989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "3e17cf7bdde24e15a361c42894205a39"
format_stamp: "Formatted at 2026-09-26 20:46:10 on dist-test-slave-4hd3"
server_key: "07c0921b0bdf4c267abb3f74e5fd2234"
server_key_iv: "b6f548db38b26910ec1eb37fbdd1338b"
server_key_version: "encryptionkey@0"
I20260926 20:46:10.415349 13989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:10.455461 13989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:10.457639 13989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:10.457815 13989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:10.461944 13989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:44399
I20260926 20:46:10.461985 14056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:44399 every 8 connection(s)
I20260926 20:46:10.462374 13989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:46:10.462589 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 13989
I20260926 20:46:10.462705 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:46:10.462970 393 external_mini_cluster.cc:1428] Setting key 2deab83121f5660c5091155ecfd7081e
I20260926 20:46:10.463447 14057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:10.466140 14057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39: Bootstrap starting.
I20260926 20:46:10.466750 14057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:10.467039 14057 log.cc:824] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:10.467809 14057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39: No bootstrap required, opened a new log
Sep 26 20:46:10 dist-test-slave-4hd3 krb5kdc[13976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:10.470126 14057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e17cf7bdde24e15a361c42894205a39" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44399 } }
I20260926 20:46:10.470304 14057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:10.470335 14057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e17cf7bdde24e15a361c42894205a39, State: Initialized, Role: FOLLOWER
I20260926 20:46:10.470476 14057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e17cf7bdde24e15a361c42894205a39" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44399 } }
I20260926 20:46:10.470559 14057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:46:10.470592 14057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:46:10.470631 14057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:10.471249 14057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e17cf7bdde24e15a361c42894205a39" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44399 } }
I20260926 20:46:10.471360 14057 leader_election.cc:302] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e17cf7bdde24e15a361c42894205a39; no voters:
I20260926 20:46:10.471539 14057 leader_election.cc:288] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:46:10.471598 14062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:46:10.471783 14062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39 [term 1 LEADER]: Becoming Leader. State: Replica: 3e17cf7bdde24e15a361c42894205a39, State: Running, Role: LEADER
I20260926 20:46:10.471848 14057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:46:10.471871 14062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e17cf7bdde24e15a361c42894205a39" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44399 } }
I20260926 20:46:10.472191 14062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e17cf7bdde24e15a361c42894205a39. Latest consensus state: current_term: 1 leader_uuid: "3e17cf7bdde24e15a361c42894205a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e17cf7bdde24e15a361c42894205a39" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44399 } } }
I20260926 20:46:10.472266 14062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39 [sys.catalog]: This master's current role is: LEADER
I20260926 20:46:10.472424 14063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e17cf7bdde24e15a361c42894205a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e17cf7bdde24e15a361c42894205a39" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44399 } } }
I20260926 20:46:10.472496 14063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39 [sys.catalog]: This master's current role is: LEADER
I20260926 20:46:10.472620 14068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:46:10.473337 14068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:46:10.474282 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:10.463497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.0.98.126:44399)
0926 20:46:10.463671 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:10.463680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:10.463705 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:46:10.464202 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:10.464209 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:10.464256 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:10.464406 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:10.465038 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:10.465929 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:10.466775 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:10.467372 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:10.467483 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:10.470234 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:10.470238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:10.470238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:10.470248 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:10.472366 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:10.472785 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:10.472788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:10.472788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:10.472935 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:10.473241 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:10.473243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:10.473243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:10.473715 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:10.473863 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:10.473982 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 20:46:10.474117 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260926 20:46:10.475178 14068 catalog_manager.cc:1347] Generated new cluster ID: 77b64c51cddf483bac892d65c5c0617b
I20260926 20:46:10.475229 14068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:46:10.475224 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:46:10.493326 14068 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:46:10.493852 14068 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:46:10.497007 14068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e17cf7bdde24e15a361c42894205a39: Generated new TSK 0
I20260926 20:46:10.497182 14068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:46:10.529030 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 13989
2026-09-26T20:46:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1346 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:46:10 dist-test-slave-4hd3 krb5kdc[14084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:46:10 dist-test-slave-4hd3 krb5kdc[14084](info): set up 2 sockets
Sep 26 20:46:10 dist-test-slave-4hd3 krb5kdc[14084](info): commencing operation
krb5kdc: starting...
W20260926 20:46:11.563875 393 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:46:11 dist-test-slave-4hd3 krb5kdc[14084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:46:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:46:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:46:11.677802 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:32789
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:32789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:46:11.758881 14097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:11.759080 14097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:11.759101 14097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:11.760337 14097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:46:11.760380 14097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:11.760406 14097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:11.760418 14097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:46:11.760454 14097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:46:11.761968 14097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:32789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:32789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:11.762207 14097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:11.762423 14097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:11.767340 14102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:11.767379 14105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:11.767531 14097 server_base.cc:1034] running on GCE node
W20260926 20:46:11.767459 14103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:11.767895 14097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:11.768150 14097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:11.769286 14097 hybrid_clock.cc:648] HybridClock initialized: now 1790455571769274 us; error 32 us; skew 500 ppm
Sep 26 20:46:11 dist-test-slave-4hd3 krb5kdc[14084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:11.771857 14097 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:46:11.772271 14097 webserver.cc:460] Webserver started at http://127.0.98.126:32827/ using document root <none> and password file <none>
I20260926 20:46:11.772488 14097 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:11.772532 14097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:11.772670 14097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:46:11.773727 14097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "92623d04bd134ee99a6ec8aa03f0883f"
format_stamp: "Formatted at 2026-09-26 20:46:11 on dist-test-slave-4hd3"
server_key: "8dc0a21520b982342a09ac8f00343119"
server_key_iv: "d32d51a445225d99dab7fb02182ec9c1"
server_key_version: "encryptionkey@0"
I20260926 20:46:11.774087 14097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "92623d04bd134ee99a6ec8aa03f0883f"
format_stamp: "Formatted at 2026-09-26 20:46:11 on dist-test-slave-4hd3"
server_key: "8dc0a21520b982342a09ac8f00343119"
server_key_iv: "d32d51a445225d99dab7fb02182ec9c1"
server_key_version: "encryptionkey@0"
I20260926 20:46:11.775367 14097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:46:11.776207 14112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:11.776400 14097 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:46:11.776489 14097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "92623d04bd134ee99a6ec8aa03f0883f"
format_stamp: "Formatted at 2026-09-26 20:46:11 on dist-test-slave-4hd3"
server_key: "8dc0a21520b982342a09ac8f00343119"
server_key_iv: "d32d51a445225d99dab7fb02182ec9c1"
server_key_version: "encryptionkey@0"
I20260926 20:46:11.776561 14097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:11.800428 14097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:11.802778 14097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:11.802942 14097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:11.806761 14097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:32789
I20260926 20:46:11.806829 14164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:32789 every 8 connection(s)
I20260926 20:46:11.807170 14097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:46:11.808154 14165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:11.810407 14165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f: Bootstrap starting.
I20260926 20:46:11.810971 14165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:11.811245 14165 log.cc:824] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:11.811993 14165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f: No bootstrap required, opened a new log
I20260926 20:46:11.812487 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 14097
I20260926 20:46:11.812601 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:46:11.812789 393 external_mini_cluster.cc:1428] Setting key a7ea883f0a93a81e002386a52a1e1b33
I20260926 20:46:11.814440 14165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92623d04bd134ee99a6ec8aa03f0883f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32789 } }
I20260926 20:46:11.814627 14165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:11.814663 14165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92623d04bd134ee99a6ec8aa03f0883f, State: Initialized, Role: FOLLOWER
I20260926 20:46:11.814793 14165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92623d04bd134ee99a6ec8aa03f0883f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32789 } }
I20260926 20:46:11.814911 14165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:46:11.814955 14165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:46:11.815006 14165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:11.815510 14165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92623d04bd134ee99a6ec8aa03f0883f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32789 } }
I20260926 20:46:11.815626 14165 leader_election.cc:302] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92623d04bd134ee99a6ec8aa03f0883f; no voters:
I20260926 20:46:11.815835 14165 leader_election.cc:288] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:46:11.815853 14170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:46:11.815977 14170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 1 LEADER]: Becoming Leader. State: Replica: 92623d04bd134ee99a6ec8aa03f0883f, State: Running, Role: LEADER
I20260926 20:46:11.816058 14170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92623d04bd134ee99a6ec8aa03f0883f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32789 } }
I20260926 20:46:11.816089 14165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:46:11.816370 14170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92623d04bd134ee99a6ec8aa03f0883f. Latest consensus state: current_term: 1 leader_uuid: "92623d04bd134ee99a6ec8aa03f0883f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92623d04bd134ee99a6ec8aa03f0883f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32789 } } }
I20260926 20:46:11.816443 14170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [sys.catalog]: This master's current role is: LEADER
I20260926 20:46:11.816377 14171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92623d04bd134ee99a6ec8aa03f0883f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92623d04bd134ee99a6ec8aa03f0883f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32789 } } }
I20260926 20:46:11.816479 14171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [sys.catalog]: This master's current role is: LEADER
W20260926 20:46:11.817816 14186 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:46:11.817868 14186 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:46:11.817920 14174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:46:11.818063 14174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:46:11.820437 14174 catalog_manager.cc:1347] Generated new cluster ID: 1184b4eddc464e89997d9b964053ba12
Sep 26 20:46:11 dist-test-slave-4hd3 krb5kdc[14084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:11.820492 14174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:46:11.824647 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:11.813278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44728 (local address 127.0.98.126:32789)
0926 20:46:11.813487 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:11.813503 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:11.813527 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:46:11.813931 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:11.813938 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:11.813982 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:11.814089 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:11.814844 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:11.816225 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:11.817395 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:11.818087 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:11.818238 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:11.820973 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:11.820977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:11.820978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:11.820989 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:11.822887 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:11.823252 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:11.823255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:11.823255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:11.823346 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:11.823524 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:11.823525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:11.823525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:11.823808 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:11.823949 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:11.824340 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 20:46:11.824461 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":134,"threads_started":1}
I20260926 20:46:11.843748 14174 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:46:11.844280 14174 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:46:11.848512 14174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f: Generated new TSK 0
I20260926 20:46:11.848701 14174 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260926 20:46:11.880298 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:32789
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:46:11.984186 14191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:11.984395 14191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:11.984418 14191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:11.985587 14191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:11.985628 14191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:11.985654 14191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:46:11.987028 14191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:32789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:11.987217 14191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:11.987422 14191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:11.992228 14197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:11.992199 14199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:11.992216 14196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:11.992542 14191 server_base.cc:1034] running on GCE node
I20260926 20:46:11.992733 14191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:11.992971 14191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:11.994119 14191 hybrid_clock.cc:648] HybridClock initialized: now 1790455571994105 us; error 28 us; skew 500 ppm
Sep 26 20:46:11 dist-test-slave-4hd3 krb5kdc[14084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:11.996657 14191 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:46:11.997107 14191 webserver.cc:460] Webserver started at http://127.0.98.65:45219/ using document root <none> and password file <none>
I20260926 20:46:11.997288 14191 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:11.997350 14191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:11.997444 14191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:46:11.998405 14191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6"
format_stamp: "Formatted at 2026-09-26 20:46:11 on dist-test-slave-4hd3"
server_key: "59e6a3ce73a8165e203f33a07c6c83a1"
server_key_iv: "71eeabdd8fb53fe6cc853def7c848e69"
server_key_version: "encryptionkey@0"
I20260926 20:46:11.998690 14191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6"
format_stamp: "Formatted at 2026-09-26 20:46:11 on dist-test-slave-4hd3"
server_key: "59e6a3ce73a8165e203f33a07c6c83a1"
server_key_iv: "71eeabdd8fb53fe6cc853def7c848e69"
server_key_version: "encryptionkey@0"
I20260926 20:46:11.999954 14191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:46:12.000679 14206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:12.000850 14191 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:46:12.000948 14191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6"
format_stamp: "Formatted at 2026-09-26 20:46:11 on dist-test-slave-4hd3"
server_key: "59e6a3ce73a8165e203f33a07c6c83a1"
server_key_iv: "71eeabdd8fb53fe6cc853def7c848e69"
server_key_version: "encryptionkey@0"
I20260926 20:46:12.001026 14191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:12.015651 14191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:12.017916 14191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:12.018062 14191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:12.018278 14191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:12.018570 14191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:46:12.018630 14191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:12.018671 14191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:46:12.018697 14191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:12.024542 14191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:43175
I20260926 20:46:12.024585 14319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:43175 every 8 connection(s)
I20260926 20:46:12.024891 14191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:46:12.025539 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 14191
I20260926 20:46:12.025642 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:46:12.025851 393 external_mini_cluster.cc:1428] Setting key 73cc89e459823c740a15198a5646a98b
Sep 26 20:46:12 dist-test-slave-4hd3 krb5kdc[14084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260926 20:46:12.035460 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.025581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:37223 (local address 127.0.98.126:32789)
0926 20:46:12.025696 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:12.025699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:12.025904 (+ 205us) server_negotiation.cc:407] Connection header received
0926 20:46:12.026595 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:12.026596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:12.026608 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:12.026666 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:12.027593 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.028211 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.029412 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.029783 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.029863 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.032092 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:12.032094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:12.032094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:12.032103 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:12.033895 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:12.034325 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:12.034326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:12.034327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:12.034416 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:12.034723 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:12.034724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:12.034725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:12.034898 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:12.034962 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:12.035264 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 20:46:12.035357 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 20:46:12.035616 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.025553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:37223)
0926 20:46:12.025765 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:12.025806 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:12.026470 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:12.026734 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:12.026738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:12.027304 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:12.027497 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.027502 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.028302 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.028304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.029321 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.029324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.029898 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.029900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.030007 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.030584 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:12.030592 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:12.031996 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:12.033998 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:12.034006 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:12.034008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:12.034251 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:12.034519 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:12.034520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:12.034520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:12.034630 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:12.034989 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:12.035008 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:12.035135 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 20:46:12.035228 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 20:46:12.035396 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":93,"threads_started":1}
I20260926 20:46:12.036253 14320 heartbeater.cc:347] Connected to a master server at 127.0.98.126:32789
I20260926 20:46:12.036420 14320 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260926 20:46:12.036626 14320 heartbeater.cc:511] Master 127.0.98.126:32789 requested a full tablet report, sending...
I20260926 20:46:12.037236 14129 ts_manager.cc:194] Registered new tserver with Master: c50a1659eb7b4c0e92c84bc3b2039aa6 (127.0.98.65:43175)
I20260926 20:46:12.038205 14129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:37223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260926 20:46:12.050742 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:32789
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:46:12.130995 14325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:12.131224 14325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:12.131263 14325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:12.132434 14325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:12.132500 14325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:12.132545 14325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:46:12.134007 14325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:32789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:12.134261 14325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:12.134495 14325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:12.139544 14331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:12.139513 14333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:12.139554 14330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:12.139681 14325 server_base.cc:1034] running on GCE node
I20260926 20:46:12.139962 14325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:12.140220 14325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:12.141415 14325 hybrid_clock.cc:648] HybridClock initialized: now 1790455572141397 us; error 30 us; skew 500 ppm
Sep 26 20:46:12 dist-test-slave-4hd3 krb5kdc[14084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:12.143883 14325 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:46:12.144299 14325 webserver.cc:460] Webserver started at http://127.0.98.66:42243/ using document root <none> and password file <none>
I20260926 20:46:12.144537 14325 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:12.144606 14325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:12.144740 14325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:46:12.145772 14325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "f245503fb5e743f48202b2ec9414f10c"
format_stamp: "Formatted at 2026-09-26 20:46:12 on dist-test-slave-4hd3"
server_key: "72ef761155dbe98a003657845294ee47"
server_key_iv: "9fc00ee181220099a2dc15cf19c32b39"
server_key_version: "encryptionkey@0"
I20260926 20:46:12.146132 14325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "f245503fb5e743f48202b2ec9414f10c"
format_stamp: "Formatted at 2026-09-26 20:46:12 on dist-test-slave-4hd3"
server_key: "72ef761155dbe98a003657845294ee47"
server_key_iv: "9fc00ee181220099a2dc15cf19c32b39"
server_key_version: "encryptionkey@0"
I20260926 20:46:12.147541 14325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:46:12.148343 14340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:12.148525 14325 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:46:12.148622 14325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "f245503fb5e743f48202b2ec9414f10c"
format_stamp: "Formatted at 2026-09-26 20:46:12 on dist-test-slave-4hd3"
server_key: "72ef761155dbe98a003657845294ee47"
server_key_iv: "9fc00ee181220099a2dc15cf19c32b39"
server_key_version: "encryptionkey@0"
I20260926 20:46:12.148710 14325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:12.160140 14325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:12.161922 14325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:12.162081 14325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:12.162310 14325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:12.162606 14325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:46:12.162659 14325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:12.162700 14325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:46:12.162729 14325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:12.168588 14325 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:35809
I20260926 20:46:12.168628 14453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:35809 every 8 connection(s)
I20260926 20:46:12.168946 14325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:46:12 dist-test-slave-4hd3 krb5kdc[14084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:12.175429 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 14325
I20260926 20:46:12.175544 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:46:12.175809 393 external_mini_cluster.cc:1428] Setting key 58c55c3b7ff1c3a02a1c7dae78bec46d
I20260926 20:46:12.178542 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.169659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:56019 (local address 127.0.98.126:32789)
0926 20:46:12.169701 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:12.169704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:12.169872 (+ 168us) server_negotiation.cc:407] Connection header received
0926 20:46:12.170359 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:12.170360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:12.170370 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:12.170428 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:12.171328 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.171902 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.172903 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.173082 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.173131 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.175253 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:12.175255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:12.175255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:12.175264 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:12.177025 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:12.177440 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:12.177442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:12.177442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:12.177507 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:12.177836 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:12.177837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:12.177837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:12.177994 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:12.178038 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:12.178404 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 20:46:12.178475 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:46:12.178876 14456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.169578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:56019)
0926 20:46:12.169747 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:12.169790 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:12.170278 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:12.170503 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:12.170507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:12.171065 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:12.171255 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.171260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.171980 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.171980 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.172814 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.172817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.173152 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.173153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.173250 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.173707 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:12.173715 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:12.175131 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:12.177108 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:12.177114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:12.177116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:12.177357 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:12.177576 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:12.177577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:12.177577 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:12.177734 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:12.178087 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:12.178093 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:12.178258 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 20:46:12.178372 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 20:46:12.178548 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260926 20:46:12.179364 14454 heartbeater.cc:347] Connected to a master server at 127.0.98.126:32789
I20260926 20:46:12.179471 14454 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:12.179649 14454 heartbeater.cc:511] Master 127.0.98.126:32789 requested a full tablet report, sending...
I20260926 20:46:12.180086 14129 ts_manager.cc:194] Registered new tserver with Master: f245503fb5e743f48202b2ec9414f10c (127.0.98.66:35809)
I20260926 20:46:12.180459 14129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:56019
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260926 20:46:12.199703 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:32789
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:46:12.280540 14459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:12.280741 14459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:12.280764 14459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:12.281823 14459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:12.281858 14459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:12.281888 14459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:46:12.283221 14459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:32789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:12.283401 14459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:12.283598 14459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:12.288513 14467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:12.288552 14465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:12.288561 14464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:12.288801 14459 server_base.cc:1034] running on GCE node
I20260926 20:46:12.289031 14459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:12.289280 14459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:12.290464 14459 hybrid_clock.cc:648] HybridClock initialized: now 1790455572290446 us; error 31 us; skew 500 ppm
Sep 26 20:46:12 dist-test-slave-4hd3 krb5kdc[14084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:12.292979 14459 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:46:12.293362 14459 webserver.cc:460] Webserver started at http://127.0.98.67:40615/ using document root <none> and password file <none>
I20260926 20:46:12.293555 14459 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:12.293596 14459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:12.293680 14459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:46:12.294685 14459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "a62667eb63b04a238b9c97b6db2e4718"
format_stamp: "Formatted at 2026-09-26 20:46:12 on dist-test-slave-4hd3"
server_key: "47ffd9b484dbb6774fed31b0bd855864"
server_key_iv: "fe36ea92be0e692572ce8d02058e853e"
server_key_version: "encryptionkey@0"
I20260926 20:46:12.295029 14459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "a62667eb63b04a238b9c97b6db2e4718"
format_stamp: "Formatted at 2026-09-26 20:46:12 on dist-test-slave-4hd3"
server_key: "47ffd9b484dbb6774fed31b0bd855864"
server_key_iv: "fe36ea92be0e692572ce8d02058e853e"
server_key_version: "encryptionkey@0"
I20260926 20:46:12.296466 14459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:46:12.297151 14474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:12.297302 14459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:46:12.297391 14459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "a62667eb63b04a238b9c97b6db2e4718"
format_stamp: "Formatted at 2026-09-26 20:46:12 on dist-test-slave-4hd3"
server_key: "47ffd9b484dbb6774fed31b0bd855864"
server_key_iv: "fe36ea92be0e692572ce8d02058e853e"
server_key_version: "encryptionkey@0"
I20260926 20:46:12.297442 14459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:12.305667 14459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:12.307410 14459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:12.307520 14459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:12.307690 14459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:12.308007 14459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:46:12.308039 14459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:12.308059 14459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:46:12.308072 14459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:12.313817 14459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:37911
I20260926 20:46:12.313891 14587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:37911 every 8 connection(s)
I20260926 20:46:12.314209 14459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:46:12 dist-test-slave-4hd3 krb5kdc[14084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:12.322891 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.314907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:60299 (local address 127.0.98.126:32789)
0926 20:46:12.314985 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:12.314987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:12.315131 (+ 144us) server_negotiation.cc:407] Connection header received
0926 20:46:12.315606 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:12.315607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:12.315617 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:12.315712 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:12.316731 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.317279 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.318346 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.318540 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.318594 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.320658 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:12.320659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:12.320659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:12.320665 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:12.321819 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:12.322167 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:12.322168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:12.322168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:12.322215 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:12.322410 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:12.322411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:12.322411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:12.322502 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:12.322565 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:12.322775 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 20:46:12.322831 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:46:12.323127 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.314871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:60299)
0926 20:46:12.315019 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:12.315053 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:12.315526 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:12.315831 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:12.315834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:12.316423 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:12.316640 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.316650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.317432 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.317435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.318273 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.318276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.318640 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.318641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.318744 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.319257 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:12.319265 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:12.320576 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:12.321899 (+ 1323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:12.321904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:12.321905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:12.322106 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:12.322271 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:12.322272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:12.322272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:12.322360 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:12.322569 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:12.322573 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:12.322675 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 20:46:12.322753 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:46:12.322862 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20260926 20:46:12.323603 14588 heartbeater.cc:347] Connected to a master server at 127.0.98.126:32789
I20260926 20:46:12.323701 14588 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:12.323902 14588 heartbeater.cc:511] Master 127.0.98.126:32789 requested a full tablet report, sending...
I20260926 20:46:12.324302 14129 ts_manager.cc:194] Registered new tserver with Master: a62667eb63b04a238b9c97b6db2e4718 (127.0.98.67:37911)
I20260926 20:46:12.324333 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 14459
I20260926 20:46:12.324414 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:46:12.324608 393 external_mini_cluster.cc:1428] Setting key 6dd5f39eaef19c5d65c71b9a97af724e
I20260926 20:46:12.324743 14129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:60299
I20260926 20:46:12.325435 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:46:12.332091 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.326685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44730 (local address 127.0.98.126:32789)
0926 20:46:12.326732 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:12.326734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:12.326857 (+ 123us) server_negotiation.cc:407] Connection header received
0926 20:46:12.326906 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:12.326907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:12.326914 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:12.326958 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:12.327435 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.327943 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.328613 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.328780 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.328828 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.329930 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:12.329931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:12.329931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:12.329936 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:12.331083 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:12.331408 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:12.331409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:12.331409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:12.331458 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:12.331614 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:12.331614 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:12.331615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:12.331814 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:12.331892 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:12.331972 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 20:46:12.332029 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 20:46:12.333734 14129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:46:12.335340 14129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:46:12.341974 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.337645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.0.98.65:43175)
0926 20:46:12.337801 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:12.337806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:12.337817 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:46:12.337858 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:12.337860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:12.337892 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:12.337985 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:12.338511 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.339579 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.340759 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.341508 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.341593 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.341655 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:12.341840 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 20:46:12.341886 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260926 20:46:12.342000 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.337532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43175 (local address 127.0.0.1:60376)
0926 20:46:12.337648 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:12.337687 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:12.337746 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:12.338086 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:12.338090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:12.338093 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:12.338163 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.338165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.339682 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.339683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.340669 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.340672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.341623 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.341624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.341721 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.341738 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 20:46:12.341815 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:46:12.341900 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":33,"thread_start_us":73,"threads_started":1}
I20260926 20:46:12.342211 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.337956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.0.98.67:37911)
0926 20:46:12.338286 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:12.338290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:12.338300 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:46:12.338334 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:12.338337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:12.338353 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:12.338436 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:12.338611 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.339287 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.340867 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.341537 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.341600 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.341655 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:12.342093 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 20:46:12.342136 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":74,"threads_started":1}
I20260926 20:46:12.342001 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.337870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:37911 (local address 127.0.0.1:53972)
0926 20:46:12.338102 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:12.338113 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:12.338159 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:12.338515 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:12.338516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:12.338517 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:12.338542 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.338542 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.339420 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.339421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.340776 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.340779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.341622 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.341623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.341721 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.341738 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 20:46:12.341816 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:46:12.341896 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":67,"thread_start_us":56,"threads_started":1}
I20260926 20:46:12.342852 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.337747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:35809 (local address 127.0.0.1:51458)
0926 20:46:12.337970 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:12.337980 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:12.338026 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:12.338317 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:12.338317 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:12.338319 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:12.338358 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.338359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.339646 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.339647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.340785 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.340787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.342620 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.342621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.342681 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.342684 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:46:12.342732 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 20:46:12.342770 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":34,"thread_start_us":45,"threads_started":1}
I20260926 20:46:12.342875 14254 tablet_service.cc:1467] Processing CreateTablet for tablet ddbcd398456c43eaab8474faf8199df8 (DEFAULT_TABLE table=test-table [id=d55c793d3f234c03bd2207e5fef5d5cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:46:12.343046 14522 tablet_service.cc:1467] Processing CreateTablet for tablet ddbcd398456c43eaab8474faf8199df8 (DEFAULT_TABLE table=test-table [id=d55c793d3f234c03bd2207e5fef5d5cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:46:12.343163 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.337940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51458 (local address 127.0.98.66:35809)
0926 20:46:12.338148 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:12.338152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:12.338160 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:46:12.338216 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:12.338218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:12.338233 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:12.338311 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:12.338416 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.339549 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.341693 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.342462 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.342533 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.342586 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:12.343043 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 20:46:12.343087 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20260926 20:46:12.343238 14254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddbcd398456c43eaab8474faf8199df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:12.343391 14522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddbcd398456c43eaab8474faf8199df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:12.343779 14388 tablet_service.cc:1467] Processing CreateTablet for tablet ddbcd398456c43eaab8474faf8199df8 (DEFAULT_TABLE table=test-table [id=d55c793d3f234c03bd2207e5fef5d5cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:46:12.344102 14388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddbcd398456c43eaab8474faf8199df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:12.345966 14602 tablet_bootstrap.cc:492] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718: Bootstrap starting.
I20260926 20:46:12.346621 14603 tablet_bootstrap.cc:492] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6: Bootstrap starting.
I20260926 20:46:12.346666 14602 tablet_bootstrap.cc:654] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:12.347301 14604 tablet_bootstrap.cc:492] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c: Bootstrap starting.
I20260926 20:46:12.347332 14603 tablet_bootstrap.cc:654] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:12.347469 14602 log.cc:824] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:12.347980 14604 tablet_bootstrap.cc:654] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:12.347946 14603 log.cc:824] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:12.348315 14602 tablet_bootstrap.cc:492] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718: No bootstrap required, opened a new log
I20260926 20:46:12.348408 14602 ts_tablet_manager.cc:1397] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:46:12.348533 14604 log.cc:824] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:12.348726 14603 tablet_bootstrap.cc:492] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6: No bootstrap required, opened a new log
I20260926 20:46:12.348825 14603 ts_tablet_manager.cc:1397] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 20:46:12.349371 14604 tablet_bootstrap.cc:492] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c: No bootstrap required, opened a new log
I20260926 20:46:12.349458 14604 ts_tablet_manager.cc:1397] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:46:12.350379 14602 raft_consensus.cc:348] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } } peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } }
I20260926 20:46:12.350567 14602 raft_consensus.cc:374] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:12.350602 14602 raft_consensus.cc:729] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62667eb63b04a238b9c97b6db2e4718, State: Initialized, Role: FOLLOWER
I20260926 20:46:12.350757 14602 consensus_queue.cc:260] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } } peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } }
I20260926 20:46:12.350737 14603 raft_consensus.cc:348] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } } peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } }
I20260926 20:46:12.350910 14603 raft_consensus.cc:374] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:12.350950 14603 raft_consensus.cc:729] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c50a1659eb7b4c0e92c84bc3b2039aa6, State: Initialized, Role: FOLLOWER
I20260926 20:46:12.351049 14602 ts_tablet_manager.cc:1428] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 20:46:12.351063 14603 consensus_queue.cc:260] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } } peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } }
I20260926 20:46:12.351176 14588 heartbeater.cc:503] Master 127.0.98.126:32789 was elected leader, sending a full tablet report...
I20260926 20:46:12.351240 14604 raft_consensus.cc:348] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } } peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } }
I20260926 20:46:12.351394 14604 raft_consensus.cc:374] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:12.351404 14603 ts_tablet_manager.cc:1428] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 20:46:12.351438 14604 raft_consensus.cc:729] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f245503fb5e743f48202b2ec9414f10c, State: Initialized, Role: FOLLOWER
I20260926 20:46:12.351533 14320 heartbeater.cc:503] Master 127.0.98.126:32789 was elected leader, sending a full tablet report...
I20260926 20:46:12.351564 14604 consensus_queue.cc:260] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } } peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } }
I20260926 20:46:12.351842 14604 ts_tablet_manager.cc:1428] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:46:12.351958 14454 heartbeater.cc:503] Master 127.0.98.126:32789 was elected leader, sending a full tablet report...
W20260926 20:46:12.419472 14455 tablet.cc:2369] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:46:12.439915 14610 raft_consensus.cc:482] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:46:12.440043 14610 raft_consensus.cc:504] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } } peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } }
I20260926 20:46:12.440445 14610 leader_election.cc:288] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c50a1659eb7b4c0e92c84bc3b2039aa6 (127.0.98.65:43175), a62667eb63b04a238b9c97b6db2e4718 (127.0.98.67:37911)
I20260926 20:46:12.443436 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.440844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:56377 (local address 127.0.98.65:43175)
0926 20:46:12.440925 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:12.440929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:12.440942 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:46:12.440967 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:12.440967 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:12.440977 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:12.441054 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:12.441212 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.441698 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.442702 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.443079 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.443160 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.443171 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:12.443328 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 20:46:12.443364 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:46:12.443437 14456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.440641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43175 (local address 127.0.98.66:56377)
0926 20:46:12.440727 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:12.440741 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:12.440833 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:12.441060 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:12.441061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:12.441063 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:12.441136 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.441137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.441786 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.441788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.442604 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.442607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.443171 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.443172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.443251 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.443254 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:46:12.443321 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 20:46:12.443363 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":41}
I20260926 20:46:12.443504 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.440872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:33049 (local address 127.0.98.67:37911)
0926 20:46:12.440925 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:12.440927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:12.441048 (+ 121us) server_negotiation.cc:407] Connection header received
0926 20:46:12.441077 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:12.441078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:12.441096 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:12.441159 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:12.441318 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.441802 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.442756 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.443141 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.443225 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.443234 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:12.443396 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 20:46:12.443430 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 20:46:12.443576 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.440759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:37911 (local address 127.0.98.66:33049)
0926 20:46:12.440957 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:12.440966 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:12.441014 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:12.441164 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:12.441165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:12.441167 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:12.441238 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.441239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.441893 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.441894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.442663 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.442665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.443229 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.443230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.443305 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.443309 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:46:12.443388 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:46:12.443429 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":123,"threads_started":1}
I20260926 20:46:12.443830 14274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddbcd398456c43eaab8474faf8199df8" candidate_uuid: "f245503fb5e743f48202b2ec9414f10c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" is_pre_election: true
I20260926 20:46:12.444029 14274 raft_consensus.cc:2454] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f245503fb5e743f48202b2ec9414f10c in term 0.
I20260926 20:46:12.444083 14542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddbcd398456c43eaab8474faf8199df8" candidate_uuid: "f245503fb5e743f48202b2ec9414f10c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62667eb63b04a238b9c97b6db2e4718" is_pre_election: true
I20260926 20:46:12.444248 14542 raft_consensus.cc:2454] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f245503fb5e743f48202b2ec9414f10c in term 0.
I20260926 20:46:12.444262 14344 leader_election.cc:302] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c50a1659eb7b4c0e92c84bc3b2039aa6, f245503fb5e743f48202b2ec9414f10c; no voters:
I20260926 20:46:12.444407 14610 raft_consensus.cc:2781] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:46:12.444479 14610 raft_consensus.cc:482] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:46:12.444515 14610 raft_consensus.cc:3037] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:12.445215 14610 raft_consensus.cc:504] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } } peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } }
I20260926 20:46:12.445322 14610 leader_election.cc:288] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [CANDIDATE]: Term 1 election: Requested vote from peers c50a1659eb7b4c0e92c84bc3b2039aa6 (127.0.98.65:43175), a62667eb63b04a238b9c97b6db2e4718 (127.0.98.67:37911)
I20260926 20:46:12.445547 14542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddbcd398456c43eaab8474faf8199df8" candidate_uuid: "f245503fb5e743f48202b2ec9414f10c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62667eb63b04a238b9c97b6db2e4718"
I20260926 20:46:12.445622 14542 raft_consensus.cc:3037] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:12.446290 14542 raft_consensus.cc:2454] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f245503fb5e743f48202b2ec9414f10c in term 1.
I20260926 20:46:12.446439 14274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddbcd398456c43eaab8474faf8199df8" candidate_uuid: "f245503fb5e743f48202b2ec9414f10c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6"
I20260926 20:46:12.446470 14344 leader_election.cc:302] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a62667eb63b04a238b9c97b6db2e4718, f245503fb5e743f48202b2ec9414f10c; no voters:
I20260926 20:46:12.446535 14274 raft_consensus.cc:3037] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:12.446607 14610 raft_consensus.cc:2781] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:46:12.446771 14610 raft_consensus.cc:686] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [term 1 LEADER]: Becoming Leader. State: Replica: f245503fb5e743f48202b2ec9414f10c, State: Running, Role: LEADER
I20260926 20:46:12.446870 14610 consensus_queue.cc:237] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } } peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } }
I20260926 20:46:12.447460 14274 raft_consensus.cc:2454] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f245503fb5e743f48202b2ec9414f10c in term 1.
I20260926 20:46:12.447832 14127 catalog_manager.cc:5495] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c reported cstate change: term changed from 0 to 1, leader changed from <none> to f245503fb5e743f48202b2ec9414f10c (127.0.98.66). New cstate: current_term: 1 leader_uuid: "f245503fb5e743f48202b2ec9414f10c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } health_report { overall_health: UNKNOWN } } }
I20260926 20:46:12.460863 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.458470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51468 (local address 127.0.98.66:35809)
0926 20:46:12.458527 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:12.458529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:12.458544 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:46:12.458573 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:12.458574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:12.458593 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:12.458660 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:46:12.458804 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.459258 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.460041 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.460216 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.460269 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.460543 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:46:12.460640 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:46:12.460712 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:12.460788 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 20:46:12.460805 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 20:46:12.462633 14274 raft_consensus.cc:1261] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Refusing update from remote peer f245503fb5e743f48202b2ec9414f10c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:46:12.462646 14542 raft_consensus.cc:1261] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718 [term 1 FOLLOWER]: Refusing update from remote peer f245503fb5e743f48202b2ec9414f10c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:46:12.463039 14610 consensus_queue.cc:1035] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:46:12.463299 14612 consensus_queue.cc:1035] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:46:12.465950 14616 mvcc.cc:205] Tried to move back new op lower bound from 7333706024804143104 to 7333706024742973440. Current Snapshot: MvccSnapshot[applied={T|T < 7333706024804143104}]
I20260926 20:46:12.466552 14617 mvcc.cc:205] Tried to move back new op lower bound from 7333706024804143104 to 7333706024742973440. Current Snapshot: MvccSnapshot[applied={T|T < 7333706024804143104}]
I20260926 20:46:12.468382 14129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:46:12.468492 14129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:46:12.469226 14129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f: Sending DeleteTablet for 3 replicas of tablet ddbcd398456c43eaab8474faf8199df8
I20260926 20:46:12.469552 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 14191
I20260926 20:46:12.469533 14254 tablet_service.cc:1514] Processing DeleteTablet for tablet ddbcd398456c43eaab8474faf8199df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:46:12 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:60376
I20260926 20:46:12.469645 14522 tablet_service.cc:1514] Processing DeleteTablet for tablet ddbcd398456c43eaab8474faf8199df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:46:12 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:53972
I20260926 20:46:12.470065 14388 tablet_service.cc:1514] Processing DeleteTablet for tablet ddbcd398456c43eaab8474faf8199df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:46:12 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:51458
I20260926 20:46:12.470402 14625 tablet_replica.cc:330] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718: stopping tablet replica
I20260926 20:46:12.470503 14626 tablet_replica.cc:330] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c: stopping tablet replica
I20260926 20:46:12.470597 14625 raft_consensus.cc:2229] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:46:12.470602 14626 raft_consensus.cc:2229] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:46:12.470695 14626 raft_consensus.cc:2258] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:46:12.470803 14625 raft_consensus.cc:2258] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:46:12.471145 14626 ts_tablet_manager.cc:1905] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:46:12.471339 14625 ts_tablet_manager.cc:1905] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:46:12.472883 14626 ts_tablet_manager.cc:1918] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:46:12.472958 14626 log.cc:1196] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/ddbcd398456c43eaab8474faf8199df8
I20260926 20:46:12.473227 14626 ts_tablet_manager.cc:1939] T ddbcd398456c43eaab8474faf8199df8 P f245503fb5e743f48202b2ec9414f10c: Deleting consensus metadata
I20260926 20:46:12.473560 14625 ts_tablet_manager.cc:1918] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:46:12.473629 14116 catalog_manager.cc:4841] TS f245503fb5e743f48202b2ec9414f10c (127.0.98.66:35809): tablet ddbcd398456c43eaab8474faf8199df8 (table test-table [id=d55c793d3f234c03bd2207e5fef5d5cd]) successfully deleted
I20260926 20:46:12.473687 14625 log.cc:1196] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/ddbcd398456c43eaab8474faf8199df8
I20260926 20:46:12.474040 14625 ts_tablet_manager.cc:1939] T ddbcd398456c43eaab8474faf8199df8 P a62667eb63b04a238b9c97b6db2e4718: Deleting consensus metadata
I20260926 20:46:12.474431 14116 catalog_manager.cc:4841] TS a62667eb63b04a238b9c97b6db2e4718 (127.0.98.67:37911): tablet ddbcd398456c43eaab8474faf8199df8 (table test-table [id=d55c793d3f234c03bd2207e5fef5d5cd]) successfully deleted
W20260926 20:46:12.475354 14116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:43175 (error 108)
I20260926 20:46:12.475982 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.475758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43175 (local address 127.0.0.1:60382)
0926 20:46:12.475903 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:12.475948 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260926 20:46:12.476117 14116 catalog_manager.cc:4584] TS c50a1659eb7b4c0e92c84bc3b2039aa6 (127.0.98.65:43175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ddbcd398456c43eaab8474faf8199df8: Network error: Client connection negotiation failed: client connection to 127.0.98.65:43175: connect: Connection refused (error 111)
I20260926 20:46:12.476773 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 14325
I20260926 20:46:12.482635 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 14459
I20260926 20:46:12.487154 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 14097
I20260926 20:46:12.492283 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:32789
--webserver_interface=127.0.98.126
--webserver_port=32827
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:32789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:46:12.571839 14628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:12.572043 14628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:12.572067 14628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:12.573261 14628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:46:12.573304 14628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:12.573318 14628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:12.573331 14628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:46:12.573343 14628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:46:12.574810 14628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:32789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:32789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=32827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:12.575054 14628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:12.575305 14628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:12.583909 14637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:12.583909 14635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:12.583940 14628 server_base.cc:1034] running on GCE node
W20260926 20:46:12.583945 14634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:12.584390 14628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:12.584708 14628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:12.585877 14628 hybrid_clock.cc:648] HybridClock initialized: now 1790455572585861 us; error 42 us; skew 500 ppm
Sep 26 20:46:12 dist-test-slave-4hd3 krb5kdc[14084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:12.588476 14628 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:46:12.588963 14628 webserver.cc:460] Webserver started at http://127.0.98.126:32827/ using document root <none> and password file <none>
I20260926 20:46:12.589208 14628 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:12.589298 14628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:12.590637 14628 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:46:12.591250 14644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:12.591473 14628 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:46:12.591578 14628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "92623d04bd134ee99a6ec8aa03f0883f"
format_stamp: "Formatted at 2026-09-26 20:46:11 on dist-test-slave-4hd3"
server_key: "8dc0a21520b982342a09ac8f00343119"
server_key_iv: "d32d51a445225d99dab7fb02182ec9c1"
server_key_version: "encryptionkey@0"
I20260926 20:46:12.592043 14628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:12.609632 14628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:12.612061 14628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:12.612223 14628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:12.616313 14628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:32789
I20260926 20:46:12.616422 14696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:32789 every 8 connection(s)
I20260926 20:46:12.616781 14628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:46:12.618357 14697 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:46:12.619032 14697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f: Bootstrap starting.
I20260926 20:46:12.627008 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 14628
I20260926 20:46:12.627368 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:43175
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:32789
--webserver_port=45219
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:46:12.627556 14697 log.cc:824] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:12.629920 14697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:46:12.630218 14697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f: Bootstrap complete.
I20260926 20:46:12.632623 14697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92623d04bd134ee99a6ec8aa03f0883f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32789 } }
I20260926 20:46:12.632879 14697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92623d04bd134ee99a6ec8aa03f0883f, State: Initialized, Role: FOLLOWER
I20260926 20:46:12.633098 14697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92623d04bd134ee99a6ec8aa03f0883f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32789 } }
I20260926 20:46:12.633199 14697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:46:12.633256 14697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:46:12.633303 14697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:46:12.634299 14697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92623d04bd134ee99a6ec8aa03f0883f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32789 } }
I20260926 20:46:12.634418 14697 leader_election.cc:302] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92623d04bd134ee99a6ec8aa03f0883f; no voters:
I20260926 20:46:12.634604 14697 leader_election.cc:288] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:46:12.634672 14701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:46:12.634827 14701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 2 LEADER]: Becoming Leader. State: Replica: 92623d04bd134ee99a6ec8aa03f0883f, State: Running, Role: LEADER
I20260926 20:46:12.634876 14697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:46:12.634946 14701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92623d04bd134ee99a6ec8aa03f0883f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32789 } }
I20260926 20:46:12.635227 14701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92623d04bd134ee99a6ec8aa03f0883f. Latest consensus state: current_term: 2 leader_uuid: "92623d04bd134ee99a6ec8aa03f0883f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92623d04bd134ee99a6ec8aa03f0883f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32789 } } }
I20260926 20:46:12.635310 14701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [sys.catalog]: This master's current role is: LEADER
I20260926 20:46:12.635463 14702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "92623d04bd134ee99a6ec8aa03f0883f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92623d04bd134ee99a6ec8aa03f0883f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32789 } } }
I20260926 20:46:12.635540 14702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [sys.catalog]: This master's current role is: LEADER
I20260926 20:46:12.635787 14707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:46:12.636610 14707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:46:12.636768 14707 catalog_manager.cc:1259] Loaded cluster ID: 1184b4eddc464e89997d9b964053ba12
I20260926 20:46:12.637014 14707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:46:12.637389 14707 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 20:46:12.642717 14707 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 20:46:12.645192 14707 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:46:12.645392 14707 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f: Loaded TSK: 0
I20260926 20:46:12.645606 14707 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:46:12.711521 14700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:12.711769 14700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:12.711843 14700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:12.712997 14700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:12.713037 14700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:12.713063 14700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:46:12.714507 14700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:43175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=45219
--tserver_master_addrs=127.0.98.126:32789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:12.714747 14700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:12.714989 14700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:12.719820 14727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:12.719857 14724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:12.719925 14700 server_base.cc:1034] running on GCE node
W20260926 20:46:12.720079 14725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:12.720340 14700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:12.720664 14700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:12.721830 14700 hybrid_clock.cc:648] HybridClock initialized: now 1790455572721809 us; error 39 us; skew 500 ppm
Sep 26 20:46:12 dist-test-slave-4hd3 krb5kdc[14084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:12.731055 14700 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:46:12.731609 14700 webserver.cc:460] Webserver started at http://127.0.98.65:45219/ using document root <none> and password file <none>
I20260926 20:46:12.731920 14700 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:12.732016 14700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:12.733476 14700 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:46:12.734231 14734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:12.734439 14700 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:46:12.734527 14700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6"
format_stamp: "Formatted at 2026-09-26 20:46:11 on dist-test-slave-4hd3"
server_key: "59e6a3ce73a8165e203f33a07c6c83a1"
server_key_iv: "71eeabdd8fb53fe6cc853def7c848e69"
server_key_version: "encryptionkey@0"
I20260926 20:46:12.734920 14700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:12.748236 14700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:12.750670 14700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:12.750813 14700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:12.751080 14700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:12.751628 14741 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:46:12.753687 14700 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:46:12.753769 14700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 20:46:12.753808 14700 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:46:12.754834 14700 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:46:12.754884 14700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 20:46:12.755080 14741 tablet_bootstrap.cc:492] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6: Bootstrap starting.
I20260926 20:46:12.762230 14700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:43175
I20260926 20:46:12.762269 14848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:43175 every 8 connection(s)
I20260926 20:46:12.762609 14700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:46:12.762877 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 14700
I20260926 20:46:12.763155 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:35809
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:32789
--webserver_port=42243
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:46:12.766103 14741 log.cc:824] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:12.770143 14741 tablet_bootstrap.cc:492] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:46:12.770488 14741 tablet_bootstrap.cc:492] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6: Bootstrap complete.
I20260926 20:46:12.771812 14741 ts_tablet_manager.cc:1397] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
Sep 26 20:46:12 dist-test-slave-4hd3 krb5kdc[14084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:12.773509 14741 raft_consensus.cc:348] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } } peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } }
I20260926 20:46:12.773896 14741 raft_consensus.cc:729] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c50a1659eb7b4c0e92c84bc3b2039aa6, State: Initialized, Role: FOLLOWER
I20260926 20:46:12.774073 14741 consensus_queue.cc:260] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } } peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } }
I20260926 20:46:12.774422 14741 ts_tablet_manager.cc:1428] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 20:46:12.776852 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.765919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:40951)
0926 20:46:12.766351 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:12.766398 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:12.767155 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:12.767747 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:12.767752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:12.768052 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:12.768239 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.768245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.769422 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.769424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.770279 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.770283 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.770958 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.770960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.771098 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.771669 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:12.771678 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:12.773237 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:12.775221 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:12.775229 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:12.775230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:12.775470 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:12.775745 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:12.775746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:12.775746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:12.775862 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:12.776226 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:12.776233 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:12.776368 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 20:46:12.776476 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 20:46:12.776650 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":92,"threads_started":1}
I20260926 20:46:12.776862 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.766024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:40951 (local address 127.0.98.126:32789)
0926 20:46:12.766289 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:12.766300 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:12.766529 (+ 229us) server_negotiation.cc:407] Connection header received
0926 20:46:12.767464 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:12.767470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:12.767515 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:12.767689 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:12.768375 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.769293 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.770381 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.770823 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.770964 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.773350 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:12.773354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:12.773354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:12.773374 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:12.775116 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:12.775557 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:12.775560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:12.775561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:12.775657 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:12.775934 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:12.775935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:12.775936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:12.776151 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:12.776246 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:12.776543 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 20:46:12.776669 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":96,"threads_started":1}
I20260926 20:46:12.777923 14849 heartbeater.cc:347] Connected to a master server at 127.0.98.126:32789
I20260926 20:46:12.778031 14849 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:12.778234 14849 heartbeater.cc:511] Master 127.0.98.126:32789 requested a full tablet report, sending...
I20260926 20:46:12.778903 14661 ts_manager.cc:194] Registered new tserver with Master: c50a1659eb7b4c0e92c84bc3b2039aa6 (127.0.98.65:43175)
I20260926 20:46:12.779942 14661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:40951
I20260926 20:46:12.780798 14849 heartbeater.cc:503] Master 127.0.98.126:32789 was elected leader, sending a full tablet report...
I20260926 20:46:12.786887 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.779563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.0.98.65:43175)
0926 20:46:12.779710 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:12.779732 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:12.783946 (+ 4214us) server_negotiation.cc:407] Connection header received
0926 20:46:12.784067 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:12.784069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:12.784083 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:12.784185 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:12.784339 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.784983 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.785978 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.786478 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.786569 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.786624 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:12.786783 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 20:46:12.786821 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260926 20:46:12.786900 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.779491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43175 (local address 127.0.0.1:60398)
0926 20:46:12.783832 (+ 4341us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:12.783862 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:12.783966 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:12.784190 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:12.784193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:12.784195 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:12.784256 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.784262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.785068 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.785069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.785888 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.785890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.786569 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.786570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.786656 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.786669 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 20:46:12.786766 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 20:46:12.786823 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4316,"thread_start_us":89,"threads_started":1}
I20260926 20:46:12.787375 14783 tablet_service.cc:1514] Processing DeleteTablet for tablet ddbcd398456c43eaab8474faf8199df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:46:12 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:60398
I20260926 20:46:12.787966 14858 tablet_replica.cc:330] stopping tablet replica
I20260926 20:46:12.788080 14858 raft_consensus.cc:2229] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:46:12.788161 14858 raft_consensus.cc:2258] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:46:12.788429 14858 ts_tablet_manager.cc:1905] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:46:12.790343 14858 ts_tablet_manager.cc:1918] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:46:12.790480 14858 log.cc:1196] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/wal/wals/ddbcd398456c43eaab8474faf8199df8
I20260926 20:46:12.790736 14858 ts_tablet_manager.cc:1939] T ddbcd398456c43eaab8474faf8199df8 P c50a1659eb7b4c0e92c84bc3b2039aa6: Deleting consensus metadata
I20260926 20:46:12.791203 14648 catalog_manager.cc:4841] TS c50a1659eb7b4c0e92c84bc3b2039aa6 (127.0.98.65:43175): tablet ddbcd398456c43eaab8474faf8199df8 (table test-table [id=d55c793d3f234c03bd2207e5fef5d5cd]) successfully deleted
W20260926 20:46:12.848558 14851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:12.848764 14851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:12.848788 14851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:12.849882 14851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:12.849915 14851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:12.849941 14851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:46:12.851286 14851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:35809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=42243
--tserver_master_addrs=127.0.98.126:32789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:12.851470 14851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:12.851635 14851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:12.856680 14866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:12.856736 14863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:12.856726 14864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:12.856922 14851 server_base.cc:1034] running on GCE node
I20260926 20:46:12.857120 14851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:12.857311 14851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:12.858470 14851 hybrid_clock.cc:648] HybridClock initialized: now 1790455572858457 us; error 28 us; skew 500 ppm
Sep 26 20:46:12 dist-test-slave-4hd3 krb5kdc[14084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:12.861080 14851 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:46:12.861477 14851 webserver.cc:460] Webserver started at http://127.0.98.66:42243/ using document root <none> and password file <none>
I20260926 20:46:12.861701 14851 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:12.861761 14851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:12.863174 14851 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:46:12.863911 14873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:12.864100 14851 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:46:12.864200 14851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "f245503fb5e743f48202b2ec9414f10c"
format_stamp: "Formatted at 2026-09-26 20:46:12 on dist-test-slave-4hd3"
server_key: "72ef761155dbe98a003657845294ee47"
server_key_iv: "9fc00ee181220099a2dc15cf19c32b39"
server_key_version: "encryptionkey@0"
I20260926 20:46:12.864559 14851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:12.874365 14851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:12.876801 14851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:12.876965 14851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:12.877190 14851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:12.877524 14851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:46:12.877579 14851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:12.877619 14851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:46:12.877647 14851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:12.883383 14851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:35809
I20260926 20:46:12.883462 14986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:35809 every 8 connection(s)
I20260926 20:46:12.883859 14851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:46:12.888413 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 14851
I20260926 20:46:12.888684 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:37911
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:32789
--webserver_port=40615
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:46:12 dist-test-slave-4hd3 krb5kdc[14084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:12.893399 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.884566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:54149 (local address 127.0.98.126:32789)
0926 20:46:12.884651 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:12.884654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:12.884854 (+ 200us) server_negotiation.cc:407] Connection header received
0926 20:46:12.885317 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:12.885318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:12.885332 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:12.885419 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:12.886393 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.886888 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.887966 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.888185 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.888293 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.890474 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:12.890476 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:12.890476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:12.890484 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:12.891997 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:12.892426 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:12.892427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:12.892428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:12.892488 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:12.892744 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:12.892746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:12.892746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:12.892861 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:12.892939 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:12.893210 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 20:46:12.893300 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:46:12.893559 14989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:12.884546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:54149)
0926 20:46:12.884732 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:12.884766 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:12.885220 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:12.885504 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:12.885507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:12.886080 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:12.886264 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.886269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.886968 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.886969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.887886 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:12.887889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:12.888300 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:12.888301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:12.888438 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:12.889024 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:12.889033 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:12.890372 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:12.892110 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:12.892117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:12.892119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:12.892344 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:12.892556 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:12.892557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:12.892557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:12.892670 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:12.892930 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:12.892936 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:12.893076 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 20:46:12.893179 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 20:46:12.893332 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":106,"threads_started":1}
I20260926 20:46:12.894057 14987 heartbeater.cc:347] Connected to a master server at 127.0.98.126:32789
I20260926 20:46:12.894141 14987 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:12.894332 14987 heartbeater.cc:511] Master 127.0.98.126:32789 requested a full tablet report, sending...
I20260926 20:46:12.894733 14661 ts_manager.cc:194] Registered new tserver with Master: f245503fb5e743f48202b2ec9414f10c (127.0.98.66:35809)
I20260926 20:46:12.895288 14661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:54149
W20260926 20:46:12.975028 14990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:12.975243 14990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:12.975267 14990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:12.976492 14990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:12.976539 14990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:12.976567 14990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:46:12.978528 14990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:37911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=40615
--tserver_master_addrs=127.0.98.126:32789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:12.978806 14990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:12.979034 14990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:12.984053 14998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:12.984062 14996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:12.984237 14990 server_base.cc:1034] running on GCE node
W20260926 20:46:12.984372 14995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:12.984719 14990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:12.985049 14990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:12.986200 14990 hybrid_clock.cc:648] HybridClock initialized: now 1790455572986186 us; error 33 us; skew 500 ppm
Sep 26 20:46:12 dist-test-slave-4hd3 krb5kdc[14084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:12.988668 14990 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:46:12.989141 14990 webserver.cc:460] Webserver started at http://127.0.98.67:40615/ using document root <none> and password file <none>
I20260926 20:46:12.989362 14990 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:12.989437 14990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:12.990733 14990 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:46:12.991290 15005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:12.991463 14990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:46:12.991557 14990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "a62667eb63b04a238b9c97b6db2e4718"
format_stamp: "Formatted at 2026-09-26 20:46:12 on dist-test-slave-4hd3"
server_key: "47ffd9b484dbb6774fed31b0bd855864"
server_key_iv: "fe36ea92be0e692572ce8d02058e853e"
server_key_version: "encryptionkey@0"
I20260926 20:46:12.991959 14990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:13.019428 14990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:13.021557 14990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:13.021723 14990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:13.021965 14990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:13.022271 14990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:46:13.022323 14990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:13.022365 14990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:46:13.022393 14990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:13.028621 14990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:37911
I20260926 20:46:13.028681 15118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:37911 every 8 connection(s)
I20260926 20:46:13.029047 14990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:46:13.033790 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 14990
Sep 26 20:46:13 dist-test-slave-4hd3 krb5kdc[14084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:13.039369 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.029962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:42357 (local address 127.0.98.126:32789)
0926 20:46:13.030033 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:13.030035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:13.030227 (+ 192us) server_negotiation.cc:407] Connection header received
0926 20:46:13.030776 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:13.030778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:13.030791 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:13.030844 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:13.031834 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.032329 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.033354 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.033562 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.033643 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:13.036257 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:13.036259 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:13.036259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:13.036268 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:13.037880 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:13.038366 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:13.038367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:13.038368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:13.038436 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:13.038696 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:13.038696 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:13.038697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:13.038808 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:13.038881 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:13.039189 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 20:46:13.039274 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:46:13.039592 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.029942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:42357)
0926 20:46:13.030099 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:13.030135 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:13.030663 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:13.030944 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:13.030947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:13.031516 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:13.031707 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:13.031735 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.032449 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.032450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:13.033257 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:13.033259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.033666 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.033667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:13.033788 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:13.034436 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:13.034446 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:13.036165 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:13.037977 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:13.037984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:13.037986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:13.038277 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:13.038506 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:13.038507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:13.038508 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:13.038627 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:13.038902 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:13.038908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:13.039049 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 20:46:13.039169 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 20:46:13.039333 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260926 20:46:13.040058 15119 heartbeater.cc:347] Connected to a master server at 127.0.98.126:32789
I20260926 20:46:13.040180 15119 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:13.040359 15119 heartbeater.cc:511] Master 127.0.98.126:32789 requested a full tablet report, sending...
I20260926 20:46:13.040750 14661 ts_manager.cc:194] Registered new tserver with Master: a62667eb63b04a238b9c97b6db2e4718 (127.0.98.67:37911)
I20260926 20:46:13.041118 14661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:42357
I20260926 20:46:13.041301 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.034216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44732 (local address 127.0.98.126:32789)
0926 20:46:13.034350 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:13.034352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:13.034361 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:46:13.034425 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:13.034425 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:13.034434 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:13.034481 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:13.035246 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.035827 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.036645 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.036855 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.036930 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:13.038280 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:13.038282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:13.038282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:13.038289 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:13.040123 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:13.040486 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:13.040487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:13.040488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:13.040555 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:13.040752 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:13.040753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:13.040753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:13.040896 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:13.041081 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:13.041120 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 20:46:13.041198 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260926 20:46:13.042030 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:46:13.044827 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.042317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44738 (local address 127.0.98.126:32789)
0926 20:46:13.042379 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:13.042381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:13.042475 (+ 94us) server_negotiation.cc:407] Connection header received
0926 20:46:13.042557 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:13.042558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:13.042591 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:13.042671 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:46:13.042795 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.043244 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.044058 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.044266 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.044348 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:13.044509 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:46:13.044606 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:46:13.044694 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:13.044735 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 20:46:13.044757 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:46:13.045075 14661 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:46:13.046545 14661 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:46:13.048743 14783 tablet_service.cc:1467] Processing CreateTablet for tablet c819686ed1cc4877ae47d33e69351ab4 (DEFAULT_TABLE table=test-table [id=ddf7e7c5553e4cac95c63ce58a16000b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:46:13.048944 14783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c819686ed1cc4877ae47d33e69351ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:13.050525 14741 tablet_bootstrap.cc:492] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6: Bootstrap starting.
I20260926 20:46:13.050858 14741 tablet_bootstrap.cc:654] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:13.051458 14741 tablet_bootstrap.cc:492] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6: No bootstrap required, opened a new log
I20260926 20:46:13.051514 14741 ts_tablet_manager.cc:1397] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 20:46:13.051612 14741 raft_consensus.cc:348] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } } peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } }
I20260926 20:46:13.051678 14741 raft_consensus.cc:374] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:13.051744 14741 raft_consensus.cc:729] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c50a1659eb7b4c0e92c84bc3b2039aa6, State: Initialized, Role: FOLLOWER
I20260926 20:46:13.051793 14741 consensus_queue.cc:260] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } } peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } }
I20260926 20:46:13.051879 14741 ts_tablet_manager.cc:1428] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:46:13.052398 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.048259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:35809 (local address 127.0.0.1:51484)
0926 20:46:13.048311 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:13.048323 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:13.048369 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:13.048961 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:13.048962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:13.048964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:13.049001 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:13.049002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.049845 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.049846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:13.050602 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:13.050605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.052162 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.052163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:13.052220 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:13.052224 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:46:13.052282 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:46:13.052325 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 20:46:13.052520 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.048344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:37911 (local address 127.0.0.1:53982)
0926 20:46:13.048488 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:13.048498 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:13.048535 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:13.048986 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:13.048987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:13.048990 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:13.049030 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:13.049032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.050204 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.050205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:13.051250 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:13.051252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.052261 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.052262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:13.052331 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:13.052335 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:46:13.052383 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 20:46:13.052424 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260926 20:46:13.052551 15126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.048403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51484 (local address 127.0.98.66:35809)
0926 20:46:13.048761 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:13.048765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:13.048774 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:46:13.048805 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:13.048807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:13.048819 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:13.048961 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:13.049060 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.049744 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.051151 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.052077 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.052164 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:13.052225 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:13.052296 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 20:46:13.052334 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":94,"threads_started":1}
I20260926 20:46:13.052747 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.048592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.0.98.67:37911)
0926 20:46:13.048766 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:13.048771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:13.048781 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:46:13.048816 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:13.048818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:13.048834 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:13.048914 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:13.049108 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.050085 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.051358 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.052115 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.052177 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:13.052230 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:13.052602 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 20:46:13.052655 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260926 20:46:13.053517 15053 tablet_service.cc:1467] Processing CreateTablet for tablet c819686ed1cc4877ae47d33e69351ab4 (DEFAULT_TABLE table=test-table [id=ddf7e7c5553e4cac95c63ce58a16000b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:46:13.053622 14921 tablet_service.cc:1467] Processing CreateTablet for tablet c819686ed1cc4877ae47d33e69351ab4 (DEFAULT_TABLE table=test-table [id=ddf7e7c5553e4cac95c63ce58a16000b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:46:13.053857 15053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c819686ed1cc4877ae47d33e69351ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:13.053979 14921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c819686ed1cc4877ae47d33e69351ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:13.056695 15129 tablet_bootstrap.cc:492] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718: Bootstrap starting.
I20260926 20:46:13.056871 15130 tablet_bootstrap.cc:492] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c: Bootstrap starting.
I20260926 20:46:13.057308 15129 tablet_bootstrap.cc:654] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:13.057490 15130 tablet_bootstrap.cc:654] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:13.057857 15129 log.cc:824] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:13.057976 15130 log.cc:824] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:13.058609 15129 tablet_bootstrap.cc:492] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718: No bootstrap required, opened a new log
I20260926 20:46:13.058665 15129 ts_tablet_manager.cc:1397] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:46:13.058820 15130 tablet_bootstrap.cc:492] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c: No bootstrap required, opened a new log
I20260926 20:46:13.058904 15130 ts_tablet_manager.cc:1397] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:46:13.060148 15129 raft_consensus.cc:348] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } } peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } }
I20260926 20:46:13.060284 15129 raft_consensus.cc:374] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:13.060307 15129 raft_consensus.cc:729] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62667eb63b04a238b9c97b6db2e4718, State: Initialized, Role: FOLLOWER
I20260926 20:46:13.060428 15129 consensus_queue.cc:260] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } } peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } }
I20260926 20:46:13.060334 15130 raft_consensus.cc:348] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } } peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } }
I20260926 20:46:13.060498 15130 raft_consensus.cc:374] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:13.060544 15130 raft_consensus.cc:729] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f245503fb5e743f48202b2ec9414f10c, State: Initialized, Role: FOLLOWER
I20260926 20:46:13.060652 15130 consensus_queue.cc:260] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } } peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } }
I20260926 20:46:13.060711 15129 ts_tablet_manager.cc:1428] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:46:13.060854 15119 heartbeater.cc:503] Master 127.0.98.126:32789 was elected leader, sending a full tablet report...
I20260926 20:46:13.060935 14987 heartbeater.cc:503] Master 127.0.98.126:32789 was elected leader, sending a full tablet report...
I20260926 20:46:13.060961 15130 ts_tablet_manager.cc:1428] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260926 20:46:13.134500 14988 tablet.cc:2369] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:46:13.266121 14850 tablet.cc:2369] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:46:13.279635 15120 tablet.cc:2369] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:46:13.343408 15133 raft_consensus.cc:482] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:46:13.343554 15133 raft_consensus.cc:504] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } } peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } }
I20260926 20:46:13.343915 15133 leader_election.cc:288] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f245503fb5e743f48202b2ec9414f10c (127.0.98.66:35809), c50a1659eb7b4c0e92c84bc3b2039aa6 (127.0.98.65:43175)
I20260926 20:46:13.347187 15126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.344214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:45157 (local address 127.0.98.66:35809)
0926 20:46:13.344291 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:13.344293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:13.344306 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:46:13.344338 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:13.344339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:13.344351 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:13.344476 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:13.344685 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.345237 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.346327 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.346751 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.346859 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:13.346873 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:13.347063 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 20:46:13.347108 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:46:13.347195 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.344364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:49917 (local address 127.0.98.65:43175)
0926 20:46:13.344566 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:13.344568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:13.344575 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:46:13.344602 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:13.344603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:13.344614 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:13.344679 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:13.344872 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.345471 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.346427 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.346828 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.346902 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:13.346916 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:13.347060 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 20:46:13.347112 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
I20260926 20:46:13.347208 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.344123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:35809 (local address 127.0.98.67:45157)
0926 20:46:13.344184 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:13.344197 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:13.344273 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:13.344502 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:13.344503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:13.344506 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:13.344576 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:13.344579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.345332 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.345334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:13.346234 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:13.346237 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.346864 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.346866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:13.346982 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:13.346985 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:46:13.347055 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 20:46:13.347125 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":76}
I20260926 20:46:13.347218 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.344267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43175 (local address 127.0.98.67:49917)
0926 20:46:13.344435 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:13.344449 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:13.344482 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:13.344696 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:13.344697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:13.344699 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:13.344764 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:13.344766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.345562 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.345562 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:13.346332 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:13.346335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.346917 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.346918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:13.346975 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:13.346979 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:46:13.347054 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:46:13.347111 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":5,"thread_start_us":103,"threads_started":1}
I20260926 20:46:13.347657 14941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c819686ed1cc4877ae47d33e69351ab4" candidate_uuid: "a62667eb63b04a238b9c97b6db2e4718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f245503fb5e743f48202b2ec9414f10c" is_pre_election: true
I20260926 20:46:13.347738 14803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c819686ed1cc4877ae47d33e69351ab4" candidate_uuid: "a62667eb63b04a238b9c97b6db2e4718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" is_pre_election: true
I20260926 20:46:13.347875 14941 raft_consensus.cc:2454] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62667eb63b04a238b9c97b6db2e4718 in term 0.
I20260926 20:46:13.347916 14803 raft_consensus.cc:2454] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62667eb63b04a238b9c97b6db2e4718 in term 0.
I20260926 20:46:13.348114 15009 leader_election.cc:302] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a62667eb63b04a238b9c97b6db2e4718, f245503fb5e743f48202b2ec9414f10c; no voters:
I20260926 20:46:13.348263 15133 raft_consensus.cc:2781] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:46:13.348353 15133 raft_consensus.cc:482] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:46:13.348387 15133 raft_consensus.cc:3037] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:13.349114 15133 raft_consensus.cc:504] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } } peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } }
I20260926 20:46:13.349263 15133 leader_election.cc:288] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [CANDIDATE]: Term 1 election: Requested vote from peers f245503fb5e743f48202b2ec9414f10c (127.0.98.66:35809), c50a1659eb7b4c0e92c84bc3b2039aa6 (127.0.98.65:43175)
I20260926 20:46:13.349478 14941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c819686ed1cc4877ae47d33e69351ab4" candidate_uuid: "a62667eb63b04a238b9c97b6db2e4718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f245503fb5e743f48202b2ec9414f10c"
I20260926 20:46:13.349478 14803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c819686ed1cc4877ae47d33e69351ab4" candidate_uuid: "a62667eb63b04a238b9c97b6db2e4718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6"
I20260926 20:46:13.349562 14941 raft_consensus.cc:3037] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:13.349578 14803 raft_consensus.cc:3037] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:13.350414 14803 raft_consensus.cc:2454] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62667eb63b04a238b9c97b6db2e4718 in term 1.
I20260926 20:46:13.350414 14941 raft_consensus.cc:2454] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62667eb63b04a238b9c97b6db2e4718 in term 1.
I20260926 20:46:13.350587 15009 leader_election.cc:302] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a62667eb63b04a238b9c97b6db2e4718, c50a1659eb7b4c0e92c84bc3b2039aa6; no voters:
I20260926 20:46:13.350685 15133 raft_consensus.cc:2781] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:46:13.350829 15133 raft_consensus.cc:686] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [term 1 LEADER]: Becoming Leader. State: Replica: a62667eb63b04a238b9c97b6db2e4718, State: Running, Role: LEADER
I20260926 20:46:13.350903 15133 consensus_queue.cc:237] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } } peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } }
I20260926 20:46:13.351768 14661 catalog_manager.cc:5495] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 reported cstate change: term changed from 0 to 1, leader changed from <none> to a62667eb63b04a238b9c97b6db2e4718 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "a62667eb63b04a238b9c97b6db2e4718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } health_report { overall_health: UNKNOWN } } }
I20260926 20:46:13.406311 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.403976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59060 (local address 127.0.98.67:37911)
0926 20:46:13.404026 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:13.404029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:13.404055 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:46:13.404104 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:13.404105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:13.404126 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:13.404205 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:46:13.404313 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.404803 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.405550 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:13.405749 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:13.405826 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:13.405955 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:46:13.406063 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:46:13.406159 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:13.406177 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 20:46:13.406197 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 20:46:13.408272 14803 raft_consensus.cc:1261] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Refusing update from remote peer a62667eb63b04a238b9c97b6db2e4718: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:46:13.408272 14941 raft_consensus.cc:1261] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c [term 1 FOLLOWER]: Refusing update from remote peer a62667eb63b04a238b9c97b6db2e4718: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:46:13.408631 15137 consensus_queue.cc:1035] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:46:13.408772 15133 consensus_queue.cc:1035] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:46:13.410816 15140 mvcc.cc:205] Tried to move back new op lower bound from 7333706028677185536 to 7333706028445880320. Current Snapshot: MvccSnapshot[applied={T|T < 7333706028677185536}]
I20260926 20:46:13.411366 15142 mvcc.cc:205] Tried to move back new op lower bound from 7333706028677185536 to 7333706028445880320. Current Snapshot: MvccSnapshot[applied={T|T < 7333706028677185536}]
I20260926 20:46:13.413730 14660 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:46:13.413833 14660 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:46:13.414594 14660 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f: Sending DeleteTablet for 3 replicas of tablet c819686ed1cc4877ae47d33e69351ab4
I20260926 20:46:13.414870 14921 tablet_service.cc:1514] Processing DeleteTablet for tablet c819686ed1cc4877ae47d33e69351ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:46:13 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:51484
I20260926 20:46:13.414901 15053 tablet_service.cc:1514] Processing DeleteTablet for tablet c819686ed1cc4877ae47d33e69351ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:46:13 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:53982
I20260926 20:46:13.415185 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 14700
I20260926 20:46:13.415200 14783 tablet_service.cc:1514] Processing DeleteTablet for tablet c819686ed1cc4877ae47d33e69351ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:46:13 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:60398
I20260926 20:46:13.415861 15151 tablet_replica.cc:330] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718: stopping tablet replica
I20260926 20:46:13.415962 15151 raft_consensus.cc:2229] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:46:13.416077 15151 raft_consensus.cc:2258] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:46:13.416225 15150 tablet_replica.cc:330] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c: stopping tablet replica
I20260926 20:46:13.416308 15150 raft_consensus.cc:2229] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:46:13.416405 15150 raft_consensus.cc:2258] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:46:13.417235 15151 ts_tablet_manager.cc:1905] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:46:13.419430 15151 ts_tablet_manager.cc:1918] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:46:13.419482 15150 ts_tablet_manager.cc:1905] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:46:13.419601 15151 log.cc:1196] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/c819686ed1cc4877ae47d33e69351ab4
I20260926 20:46:13.420013 15151 ts_tablet_manager.cc:1939] T c819686ed1cc4877ae47d33e69351ab4 P a62667eb63b04a238b9c97b6db2e4718: Deleting consensus metadata
I20260926 20:46:13.420476 14648 catalog_manager.cc:4841] TS a62667eb63b04a238b9c97b6db2e4718 (127.0.98.67:37911): tablet c819686ed1cc4877ae47d33e69351ab4 (table test-table [id=ddf7e7c5553e4cac95c63ce58a16000b]) successfully deleted
I20260926 20:46:13.421553 15150 ts_tablet_manager.cc:1918] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:46:13.421629 15150 log.cc:1196] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/c819686ed1cc4877ae47d33e69351ab4
I20260926 20:46:13.421895 15150 ts_tablet_manager.cc:1939] T c819686ed1cc4877ae47d33e69351ab4 P f245503fb5e743f48202b2ec9414f10c: Deleting consensus metadata
W20260926 20:46:13.422030 14648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:43175 (error 108)
I20260926 20:46:13.422482 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.422334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43175 (local address 127.0.0.1:49714)
0926 20:46:13.422411 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:13.422444 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 20:46:13.422578 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 14851
W20260926 20:46:13.422616 14648 catalog_manager.cc:4584] TS c50a1659eb7b4c0e92c84bc3b2039aa6 (127.0.98.65:43175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c819686ed1cc4877ae47d33e69351ab4: Network error: Client connection negotiation failed: client connection to 127.0.98.65:43175: connect: Connection refused (error 111)
I20260926 20:46:13.427944 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.427827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:35809 (local address 127.0.0.1:53732)
0926 20:46:13.427882 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:13.427913 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.66:35809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 20:46:13.428084 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 14990
I20260926 20:46:13.432873 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 14628
I20260926 20:46:13.438098 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:32789
--webserver_interface=127.0.98.126
--webserver_port=32827
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:32789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:46:13.517402 15153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:13.517621 15153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:13.517657 15153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:13.518798 15153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:46:13.518864 15153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:13.518886 15153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:13.518906 15153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:46:13.518937 15153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:46:13.520396 15153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:32789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:32789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=32827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:13.520658 15153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:13.520874 15153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:13.525490 15159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:13.525509 15158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:13.525559 15161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:13.525827 15153 server_base.cc:1034] running on GCE node
I20260926 20:46:13.526105 15153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:13.526330 15153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:13.527475 15153 hybrid_clock.cc:648] HybridClock initialized: now 1790455573527459 us; error 30 us; skew 500 ppm
Sep 26 20:46:13 dist-test-slave-4hd3 krb5kdc[14084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:13.530009 15153 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:46:13.530440 15153 webserver.cc:460] Webserver started at http://127.0.98.126:32827/ using document root <none> and password file <none>
I20260926 20:46:13.530683 15153 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:13.530753 15153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:13.532226 15153 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:46:13.533013 15168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:13.533228 15153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:46:13.533346 15153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "92623d04bd134ee99a6ec8aa03f0883f"
format_stamp: "Formatted at 2026-09-26 20:46:11 on dist-test-slave-4hd3"
server_key: "8dc0a21520b982342a09ac8f00343119"
server_key_iv: "d32d51a445225d99dab7fb02182ec9c1"
server_key_version: "encryptionkey@0"
I20260926 20:46:13.533699 15153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:13.556000 15153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:13.558127 15153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:13.558287 15153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:13.562398 15153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:32789
I20260926 20:46:13.562429 15220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:32789 every 8 connection(s)
I20260926 20:46:13.562827 15153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:46:13.564399 15221 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:46:13.565074 15221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f: Bootstrap starting.
I20260926 20:46:13.572829 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 15153
I20260926 20:46:13.573133 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:43175
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:32789
--webserver_port=45219
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:46:13.573253 15221 log.cc:824] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:13.575523 15221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:46:13.575892 15221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f: Bootstrap complete.
I20260926 20:46:13.578097 15221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92623d04bd134ee99a6ec8aa03f0883f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32789 } }
I20260926 20:46:13.578337 15221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92623d04bd134ee99a6ec8aa03f0883f, State: Initialized, Role: FOLLOWER
I20260926 20:46:13.578526 15221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92623d04bd134ee99a6ec8aa03f0883f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32789 } }
I20260926 20:46:13.578625 15221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:46:13.578681 15221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:46:13.578732 15221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:46:13.579691 15221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92623d04bd134ee99a6ec8aa03f0883f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32789 } }
I20260926 20:46:13.579829 15221 leader_election.cc:302] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92623d04bd134ee99a6ec8aa03f0883f; no voters:
I20260926 20:46:13.580030 15221 leader_election.cc:288] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 20:46:13.580197 15225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 3 FOLLOWER]: Leader election won for term 3
I20260926 20:46:13.580314 15221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:46:13.580370 15225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [term 3 LEADER]: Becoming Leader. State: Replica: 92623d04bd134ee99a6ec8aa03f0883f, State: Running, Role: LEADER
I20260926 20:46:13.580461 15225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92623d04bd134ee99a6ec8aa03f0883f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32789 } }
I20260926 20:46:13.580709 15226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "92623d04bd134ee99a6ec8aa03f0883f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92623d04bd134ee99a6ec8aa03f0883f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32789 } } }
I20260926 20:46:13.580816 15226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [sys.catalog]: This master's current role is: LEADER
I20260926 20:46:13.580703 15225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92623d04bd134ee99a6ec8aa03f0883f. Latest consensus state: current_term: 3 leader_uuid: "92623d04bd134ee99a6ec8aa03f0883f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92623d04bd134ee99a6ec8aa03f0883f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 32789 } } }
I20260926 20:46:13.581055 15225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92623d04bd134ee99a6ec8aa03f0883f [sys.catalog]: This master's current role is: LEADER
I20260926 20:46:13.581128 15234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:46:13.582065 15234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:46:13.582216 15234 catalog_manager.cc:1259] Loaded cluster ID: 1184b4eddc464e89997d9b964053ba12
I20260926 20:46:13.582356 15234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 20:46:13.582772 15234 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc1e39369ad google::LogMessage::Fail() at ??:0
@ 0x7fc1e393988a google::LogMessage::SendToLog() at ??:0
@ 0x7fc1e39364de google::LogMessage::Flush() at ??:0
@ 0x7fc1e3937a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc1eb8570f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fc1eb8608b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc1e58040d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc1e57f9954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc1e665d6db start_thread at ??:0
@ 0x7fc1e243461f clone at ??:0
W20260926 20:46:13.669662 15224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:13.669847 15224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:13.669870 15224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:13.671046 15224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:13.671089 15224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:13.671115 15224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:46:13.672560 15224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:43175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=45219
--tserver_master_addrs=127.0.98.126:32789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:13.672786 15224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:13.673038 15224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:13.677917 15260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:13.678037 15258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:13.677927 15257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:13.678205 15224 server_base.cc:1034] running on GCE node
I20260926 20:46:13.678392 15224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:13.678645 15224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:13.679777 15224 hybrid_clock.cc:648] HybridClock initialized: now 1790455573679754 us; error 39 us; skew 500 ppm
Sep 26 20:46:13 dist-test-slave-4hd3 krb5kdc[14084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:13.682449 15224 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:46:13.682933 15224 webserver.cc:460] Webserver started at http://127.0.98.65:45219/ using document root <none> and password file <none>
I20260926 20:46:13.683156 15224 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:13.683224 15224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:13.684624 15224 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:46:13.685346 15267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:13.685624 15224 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:46:13.685750 15224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6"
format_stamp: "Formatted at 2026-09-26 20:46:11 on dist-test-slave-4hd3"
server_key: "59e6a3ce73a8165e203f33a07c6c83a1"
server_key_iv: "71eeabdd8fb53fe6cc853def7c848e69"
server_key_version: "encryptionkey@0"
I20260926 20:46:13.686097 15224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:13.700866 15224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:13.703166 15224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:13.703328 15224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:13.703572 15224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:13.704054 15274 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:46:13.706034 15224 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:46:13.706136 15224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 20:46:13.706202 15224 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:46:13.707115 15224 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:46:13.707175 15224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 20:46:13.707260 15274 tablet_bootstrap.cc:492] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6: Bootstrap starting.
I20260926 20:46:13.714309 15224 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:43175
I20260926 20:46:13.714670 15224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:46:13.715068 15381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:43175 every 8 connection(s)
I20260926 20:46:13.717875 15274 log.cc:824] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:13.718703 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 15224
I20260926 20:46:13.718976 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:35809
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:32789
--webserver_port=42243
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:46:13.720278 15274 tablet_bootstrap.cc:492] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:46:13.720587 15274 tablet_bootstrap.cc:492] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6: Bootstrap complete.
I20260926 20:46:13.721818 15274 ts_tablet_manager.cc:1397] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260926 20:46:13.723305 15274 raft_consensus.cc:348] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } } peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } }
I20260926 20:46:13.723627 15274 raft_consensus.cc:729] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c50a1659eb7b4c0e92c84bc3b2039aa6, State: Initialized, Role: FOLLOWER
I20260926 20:46:13.723788 15274 consensus_queue.cc:260] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } } peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } }
I20260926 20:46:13.724066 15274 ts_tablet_manager.cc:1428] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260926 20:46:13.787127 15384 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:46:13.715333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:33073)
0926 20:46:13.715521 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:13.715563 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:13.716119 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:13.786922 (+ 70803us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":122,"threads_started":1}
W20260926 20:46:13.787385 15382 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:32789 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.126:32789: Client connection negotiation failed: client connection to 127.0.98.126:32789: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:46:13.787786 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.787633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:47133)
0926 20:46:13.787687 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:13.787760 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 20:46:13.788149 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.788074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:34095)
0926 20:46:13.788113 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:13.788135 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 20:46:13.788272 15382 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:46:13.811084 15386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:13.811265 15386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:13.811288 15386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:13.812515 15386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:13.812556 15386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:13.812583 15386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:46:13.814050 15386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:35809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=42243
--tserver_master_addrs=127.0.98.126:32789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:13.814278 15386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:13.814515 15386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:13.819451 15392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:13.819509 15393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:13.819626 15395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:13.819876 15386 server_base.cc:1034] running on GCE node
I20260926 20:46:13.820096 15386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:13.820333 15386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:13.821520 15386 hybrid_clock.cc:648] HybridClock initialized: now 1790455573821503 us; error 31 us; skew 500 ppm
Sep 26 20:46:13 dist-test-slave-4hd3 krb5kdc[14084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:13.824088 15386 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:46:13.824590 15386 webserver.cc:460] Webserver started at http://127.0.98.66:42243/ using document root <none> and password file <none>
I20260926 20:46:13.824782 15386 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:13.824822 15386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:13.826113 15386 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:46:13.826660 15402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:13.826824 15386 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:13.826896 15386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "f245503fb5e743f48202b2ec9414f10c"
format_stamp: "Formatted at 2026-09-26 20:46:12 on dist-test-slave-4hd3"
server_key: "72ef761155dbe98a003657845294ee47"
server_key_iv: "9fc00ee181220099a2dc15cf19c32b39"
server_key_version: "encryptionkey@0"
I20260926 20:46:13.827237 15386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:13.838825 15386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:13.841125 15386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:13.841289 15386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:13.841536 15386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:13.841841 15386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:46:13.841876 15386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:13.841897 15386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:46:13.841943 15386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:13.847755 15386 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:35809
I20260926 20:46:13.847800 15515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:35809 every 8 connection(s)
I20260926 20:46:13.848158 15386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:46:13.849180 15518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.848817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:40575)
0926 20:46:13.848971 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:13.849041 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
W20260926 20:46:13.849592 15516 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:32789 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.126:32789: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
I20260926 20:46:13.849869 15518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.849771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:35447)
0926 20:46:13.849822 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:13.849847 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:46:13.850142 15518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.850060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:38675)
0926 20:46:13.850103 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:13.850124 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 20:46:13.850246 15516 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:46:13.853952 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 15386
I20260926 20:46:13.854210 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:37911
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:32789
--webserver_port=40615
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:46:13.942817 15519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:13.943053 15519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:13.943095 15519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:13.944347 15519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:13.944413 15519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:13.944473 15519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:46:13.945920 15519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:37911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=40615
--tserver_master_addrs=127.0.98.126:32789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:13.946183 15519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:13.946412 15519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:46:13.951706 15519 server_base.cc:1034] running on GCE node
W20260926 20:46:13.951654 15525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:13.951632 15527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:13.951661 15524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:13.952054 15519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:13.952262 15519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:13.953397 15519 hybrid_clock.cc:648] HybridClock initialized: now 1790455573953384 us; error 28 us; skew 500 ppm
Sep 26 20:46:13 dist-test-slave-4hd3 krb5kdc[14084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:13.955969 15519 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:46:13.956426 15519 webserver.cc:460] Webserver started at http://127.0.98.67:40615/ using document root <none> and password file <none>
I20260926 20:46:13.956650 15519 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:13.956697 15519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:13.958076 15519 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:46:13.958770 15534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:13.958978 15519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:46:13.959076 15519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "a62667eb63b04a238b9c97b6db2e4718"
format_stamp: "Formatted at 2026-09-26 20:46:12 on dist-test-slave-4hd3"
server_key: "47ffd9b484dbb6774fed31b0bd855864"
server_key_iv: "fe36ea92be0e692572ce8d02058e853e"
server_key_version: "encryptionkey@0"
I20260926 20:46:13.959467 15519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:13.985366 15519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:13.987514 15519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:13.987663 15519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:13.987941 15519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:13.988294 15519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:46:13.988341 15519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:13.988364 15519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:46:13.988384 15519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:13.994098 15519 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:37911
I20260926 20:46:13.994144 15647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:37911 every 8 connection(s)
I20260926 20:46:13.994480 15519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:46:13.995472 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.995112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:35629)
0926 20:46:13.995266 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:13.995332 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
W20260926 20:46:13.995936 15648 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:32789 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.126:32789: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
I20260926 20:46:13.996204 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.996109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:59623)
0926 20:46:13.996158 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:13.996182 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:46:13.996474 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:13.996388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:46729)
0926 20:46:13.996432 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:13.996454 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 20:46:13.996577 15648 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:46:13.999215 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 15519
I20260926 20:46:14.788837 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:14.788590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:40573)
0926 20:46:14.788705 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:14.788765 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260926 20:46:14.850782 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:14.850556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:51223)
0926 20:46:14.850699 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:14.850733 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":55,"threads_started":1}
I20260926 20:46:14.997201 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:14.996923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:41923)
0926 20:46:14.997120 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:14.997154 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":71,"threads_started":1}
I20260926 20:46:15.435005 15655 raft_consensus.cc:482] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:46:15.435135 15655 raft_consensus.cc:504] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } } peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } }
I20260926 20:46:15.435421 15655 leader_election.cc:288] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f245503fb5e743f48202b2ec9414f10c (127.0.98.66:35809), a62667eb63b04a238b9c97b6db2e4718 (127.0.98.67:37911)
Sep 26 20:46:15 dist-test-slave-4hd3 krb5kdc[14084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.67@KRBTEST.COM
Sep 26 20:46:15 dist-test-slave-4hd3 krb5kdc[14084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.66@KRBTEST.COM
I20260926 20:46:15.447387 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:15.435745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:49795 (local address 127.0.98.66:35809)
0926 20:46:15.435942 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:15.435957 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:15.435985 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:46:15.436489 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:15.436492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:15.436537 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:15.436654 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:15.437482 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:15.438457 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:15.439975 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:15.440610 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:15.440731 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:15.443940 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:15.443943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:15.443944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:15.443954 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:15.445885 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:15.446294 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:15.446296 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:15.446297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:15.446420 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:15.446610 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:15.446611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:15.446611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:15.446895 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:15.446949 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:15.447135 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 20:46:15.447269 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":74,"threads_started":1}
I20260926 20:46:15.447381 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:15.435938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:33503 (local address 127.0.98.67:37911)
0926 20:46:15.436089 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:15.436098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:15.436128 (+ 30us) server_negotiation.cc:407] Connection header received
0926 20:46:15.436575 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:15.436578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:15.436645 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:15.436794 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:15.437483 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:15.438546 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:15.439975 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:15.440615 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:15.440731 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:15.443499 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:15.443504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:15.443504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:15.443516 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:15.445545 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:15.445954 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:15.445957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:15.445957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:15.446067 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:15.446342 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:15.446343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:15.446344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:15.446716 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:15.446796 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:15.447135 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 20:46:15.447269 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
I20260926 20:46:15.447402 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:15.435810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:37911 (local address 127.0.98.65:33503)
0926 20:46:15.435918 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:15.435929 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:15.435983 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:15.436789 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:15.436790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:15.437149 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:15.437329 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:15.437335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:15.438624 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:15.438625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:15.439880 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:15.439884 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:15.440720 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:15.440721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:15.440880 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:15.441593 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:15.441603 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:15.443373 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:15.445636 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:15.445641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:15.445643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:15.445864 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:15.446133 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:15.446133 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:15.446134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:15.446269 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:15.446782 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:15.446787 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:15.446946 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 20:46:15.447064 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 20:46:15.447263 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":55,"thread_start_us":42,"threads_started":1}
I20260926 20:46:15.447752 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:15.435608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:35809 (local address 127.0.98.65:49795)
0926 20:46:15.435769 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:15.435807 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:15.435891 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:15.436729 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:15.436733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:15.437152 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:15.437329 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:15.437335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:15.438605 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:15.438606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:15.439881 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:15.439884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:15.440720 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:15.440721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:15.440878 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:15.441593 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:15.441603 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:15.443814 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:15.445991 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:15.445991 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:15.445991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:15.446209 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:15.446493 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:15.446494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:15.446494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:15.446542 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:15.446965 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:15.446965 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:15.446998 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 20:46:15.447064 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 20:46:15.447315 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":189,"thread_start_us":61,"threads_started":1}
W20260926 20:46:15.448774 15271 leader_election.cc:341] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f245503fb5e743f48202b2ec9414f10c (127.0.98.66:35809): Not found: Tablet not found: c819686ed1cc4877ae47d33e69351ab4
W20260926 20:46:15.448866 15271 leader_election.cc:341] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a62667eb63b04a238b9c97b6db2e4718 (127.0.98.67:37911): Not found: Tablet not found: c819686ed1cc4877ae47d33e69351ab4
I20260926 20:46:15.448923 15271 leader_election.cc:302] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c50a1659eb7b4c0e92c84bc3b2039aa6; no voters: a62667eb63b04a238b9c97b6db2e4718, f245503fb5e743f48202b2ec9414f10c
I20260926 20:46:15.449035 15655 raft_consensus.cc:2726] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:46:15.789513 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:15.789349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:44069)
0926 20:46:15.789427 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:15.789472 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 20:46:15.851581 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:15.851310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:38737)
0926 20:46:15.851473 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:15.851533 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
I20260926 20:46:15.997857 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:15.997643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:41155)
0926 20:46:15.997781 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:15.997816 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260926 20:46:16.790169 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:16.789955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:34177)
0926 20:46:16.790081 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:16.790120 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260926 20:46:16.852454 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:16.852142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:56263)
0926 20:46:16.852296 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:16.852377 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":61,"threads_started":1}
I20260926 20:46:16.993520 15664 raft_consensus.cc:482] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:46:16.993649 15664 raft_consensus.cc:504] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } } peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } }
I20260926 20:46:16.993847 15664 leader_election.cc:288] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f245503fb5e743f48202b2ec9414f10c (127.0.98.66:35809), a62667eb63b04a238b9c97b6db2e4718 (127.0.98.67:37911)
W20260926 20:46:16.994400 15271 leader_election.cc:341] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f245503fb5e743f48202b2ec9414f10c (127.0.98.66:35809): Not found: Tablet not found: c819686ed1cc4877ae47d33e69351ab4
W20260926 20:46:16.994992 15271 leader_election.cc:341] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a62667eb63b04a238b9c97b6db2e4718 (127.0.98.67:37911): Not found: Tablet not found: c819686ed1cc4877ae47d33e69351ab4
I20260926 20:46:16.995051 15271 leader_election.cc:302] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c50a1659eb7b4c0e92c84bc3b2039aa6; no voters: a62667eb63b04a238b9c97b6db2e4718, f245503fb5e743f48202b2ec9414f10c
I20260926 20:46:16.995178 15664 raft_consensus.cc:2726] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:46:16.998522 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:16.998320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:41195)
0926 20:46:16.998438 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:16.998474 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260926 20:46:17.790890 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:17.790666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:40449)
0926 20:46:17.790802 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:17.790841 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260926 20:46:17.853163 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:17.852962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:54085)
0926 20:46:17.853084 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:17.853121 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260926 20:46:17.999185 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:17.998981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:40169)
0926 20:46:17.999109 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:17.999145 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
I20260926 20:46:18.791780 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:18.791479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:55877)
0926 20:46:18.791632 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:18.791671 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":63,"threads_started":1}
I20260926 20:46:18.853863 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:18.853666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:42771)
0926 20:46:18.853791 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:18.853824 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260926 20:46:18.999889 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:18.999639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:42719)
0926 20:46:18.999813 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:18.999850 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":57,"threads_started":1}
I20260926 20:46:19.377663 15672 raft_consensus.cc:482] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:46:19.377758 15672 raft_consensus.cc:504] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } } peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } }
I20260926 20:46:19.377919 15672 leader_election.cc:288] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f245503fb5e743f48202b2ec9414f10c (127.0.98.66:35809), a62667eb63b04a238b9c97b6db2e4718 (127.0.98.67:37911)
W20260926 20:46:19.378307 15271 leader_election.cc:341] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f245503fb5e743f48202b2ec9414f10c (127.0.98.66:35809): Not found: Tablet not found: c819686ed1cc4877ae47d33e69351ab4
W20260926 20:46:19.378379 15271 leader_election.cc:341] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a62667eb63b04a238b9c97b6db2e4718 (127.0.98.67:37911): Not found: Tablet not found: c819686ed1cc4877ae47d33e69351ab4
I20260926 20:46:19.378394 15271 leader_election.cc:302] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c50a1659eb7b4c0e92c84bc3b2039aa6; no voters: a62667eb63b04a238b9c97b6db2e4718, f245503fb5e743f48202b2ec9414f10c
I20260926 20:46:19.378465 15672 raft_consensus.cc:2726] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:46:19.792444 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:19.792235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:58829)
0926 20:46:19.792364 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:19.792397 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260926 20:46:19.854524 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:19.854314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:49185)
0926 20:46:19.854446 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:19.854483 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260926 20:46:20.000495 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:20.000311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:34821)
0926 20:46:20.000423 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:20.000452 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260926 20:46:20.793099 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:20.792891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:45597)
0926 20:46:20.793017 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:20.793053 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260926 20:46:20.855165 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:20.854951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:51113)
0926 20:46:20.855083 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:20.855122 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260926 20:46:21.001163 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:21.000958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:60919)
0926 20:46:21.001087 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:21.001122 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260926 20:46:21.144397 15679 raft_consensus.cc:482] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:46:21.144524 15679 raft_consensus.cc:504] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } } peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } }
I20260926 20:46:21.144706 15679 leader_election.cc:288] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f245503fb5e743f48202b2ec9414f10c (127.0.98.66:35809), a62667eb63b04a238b9c97b6db2e4718 (127.0.98.67:37911)
W20260926 20:46:21.145167 15271 leader_election.cc:341] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f245503fb5e743f48202b2ec9414f10c (127.0.98.66:35809): Not found: Tablet not found: c819686ed1cc4877ae47d33e69351ab4
W20260926 20:46:21.145255 15271 leader_election.cc:341] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a62667eb63b04a238b9c97b6db2e4718 (127.0.98.67:37911): Not found: Tablet not found: c819686ed1cc4877ae47d33e69351ab4
I20260926 20:46:21.145275 15271 leader_election.cc:302] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c50a1659eb7b4c0e92c84bc3b2039aa6; no voters: a62667eb63b04a238b9c97b6db2e4718, f245503fb5e743f48202b2ec9414f10c
I20260926 20:46:21.145354 15679 raft_consensus.cc:2726] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:46:21.793905 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:21.793654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:53585)
0926 20:46:21.793820 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:21.793856 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260926 20:46:21.855926 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:21.855660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:48497)
0926 20:46:21.855848 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:21.855885 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":120,"threads_started":1}
I20260926 20:46:22.001844 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:22.001622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:50923)
0926 20:46:22.001765 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:22.001802 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":60,"threads_started":1}
I20260926 20:46:22.794651 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:22.794428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:57687)
0926 20:46:22.794568 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:22.794605 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":104,"threads_started":1}
I20260926 20:46:22.856696 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:22.856455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:55577)
0926 20:46:22.856613 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:22.856652 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260926 20:46:23.002540 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:23.002314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:35925)
0926 20:46:23.002460 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:23.002497 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260926 20:46:23.795285 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:23.795058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:48389)
0926 20:46:23.795182 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:23.795225 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260926 20:46:23.857441 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:23.857211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:49739)
0926 20:46:23.857366 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:23.857398 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
I20260926 20:46:24.003254 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:24.003016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:46573)
0926 20:46:24.003168 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:24.003207 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
I20260926 20:46:24.796005 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:24.795785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:44745)
0926 20:46:24.795923 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:24.795958 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260926 20:46:24.858273 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:24.858049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:49911)
0926 20:46:24.858187 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:24.858225 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260926 20:46:25.004056 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:25.003850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:38739)
0926 20:46:25.003965 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:25.004008 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260926 20:46:25.619477 15692 raft_consensus.cc:482] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:46:25.619583 15692 raft_consensus.cc:504] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245503fb5e743f48202b2ec9414f10c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35809 } } peers { permanent_uuid: "a62667eb63b04a238b9c97b6db2e4718" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37911 } } peers { permanent_uuid: "c50a1659eb7b4c0e92c84bc3b2039aa6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43175 } }
I20260926 20:46:25.619781 15692 leader_election.cc:288] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f245503fb5e743f48202b2ec9414f10c (127.0.98.66:35809), a62667eb63b04a238b9c97b6db2e4718 (127.0.98.67:37911)
W20260926 20:46:25.620190 15271 leader_election.cc:341] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f245503fb5e743f48202b2ec9414f10c (127.0.98.66:35809): Not found: Tablet not found: c819686ed1cc4877ae47d33e69351ab4
W20260926 20:46:25.620288 15271 leader_election.cc:341] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a62667eb63b04a238b9c97b6db2e4718 (127.0.98.67:37911): Not found: Tablet not found: c819686ed1cc4877ae47d33e69351ab4
I20260926 20:46:25.620324 15271 leader_election.cc:302] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c50a1659eb7b4c0e92c84bc3b2039aa6; no voters: a62667eb63b04a238b9c97b6db2e4718, f245503fb5e743f48202b2ec9414f10c
I20260926 20:46:25.620416 15692 raft_consensus.cc:2726] T c819686ed1cc4877ae47d33e69351ab4 P c50a1659eb7b4c0e92c84bc3b2039aa6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:46:25.796692 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:25.796475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:51449)
0926 20:46:25.796613 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:25.796648 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":61,"threads_started":1}
I20260926 20:46:25.859035 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:25.858795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:45801)
0926 20:46:25.858949 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:25.858986 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":65,"threads_started":1}
I20260926 20:46:26.004814 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:26.004574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:35187)
0926 20:46:26.004735 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:26.004769 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260926 20:46:26.797376 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:26.797173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:57207)
0926 20:46:26.797295 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:26.797330 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260926 20:46:26.859809 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:26.859541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:40085)
0926 20:46:26.859666 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:26.859703 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260926 20:46:27.005538 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:27.005312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:52717)
0926 20:46:27.005454 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:27.005493 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260926 20:46:27.798202 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:27.797990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:47143)
0926 20:46:27.798113 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:27.798154 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260926 20:46:27.860498 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:27.860281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:36939)
0926 20:46:27.860414 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:27.860454 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260926 20:46:28.006228 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:28.006005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:40399)
0926 20:46:28.006141 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:28.006182 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260926 20:46:28.798933 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:28.798722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.65:46955)
0926 20:46:28.798850 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:28.798885 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260926 20:46:28.861230 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:28.861040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.66:59917)
0926 20:46:28.861155 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:28.861187 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260926 20:46:29.007030 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:29.006788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:32789 (local address 127.0.98.67:58887)
0926 20:46:29.006942 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:29.006982 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":66,"threads_started":1}
I20260926 20:46:29.007473 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 15224
I20260926 20:46:29.012085 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 15386
I20260926 20:46:29.016441 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 15519
2026-09-26T20:46:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18488 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:46:29 dist-test-slave-4hd3 krb5kdc[15707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:46:29 dist-test-slave-4hd3 krb5kdc[15707](info): set up 2 sockets
Sep 26 20:46:29 dist-test-slave-4hd3 krb5kdc[15707](info): commencing operation
krb5kdc: starting...
W20260926 20:46:30.055902 393 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:46:30 dist-test-slave-4hd3 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:46:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:46:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:46:30.170576 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38437
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:38437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:46:30.250530 15720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:30.250746 15720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:30.250769 15720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:30.251937 15720 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:46:30.251979 15720 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:46:30.251992 15720 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:46:30.252003 15720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:46:30.252017 15720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:30.252030 15720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:30.252043 15720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:46:30.252053 15720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:46:30.253479 15720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:38437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:30.253674 15720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:30.253852 15720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:30.258662 15725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:30.258679 15728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:30.258798 15726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:30.258848 15720 server_base.cc:1034] running on GCE node
I20260926 20:46:30.259188 15720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:30.259430 15720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:30.260591 15720 hybrid_clock.cc:648] HybridClock initialized: now 1790455590260583 us; error 38 us; skew 500 ppm
Sep 26 20:46:30 dist-test-slave-4hd3 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:30.263015 15720 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:46:30.263435 15720 webserver.cc:460] Webserver started at http://127.0.98.126:40097/ using document root <none> and password file <none>
I20260926 20:46:30.263674 15720 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:30.263765 15720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:30.263914 15720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:46:30.265091 15720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "a60fabb67dd24bb59ec33c8673857818"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "9e3d30525cb39c28e21c8401ce659449"
server_key_iv: "f0e826300623b9c1505d6c0b00525d3b"
server_key_version: "encryptionkey@0"
I20260926 20:46:30.265461 15720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "a60fabb67dd24bb59ec33c8673857818"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "9e3d30525cb39c28e21c8401ce659449"
server_key_iv: "f0e826300623b9c1505d6c0b00525d3b"
server_key_version: "encryptionkey@0"
I20260926 20:46:30.266798 15720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:46:30.267518 15735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:30.267693 15720 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:46:30.267822 15720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "a60fabb67dd24bb59ec33c8673857818"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "9e3d30525cb39c28e21c8401ce659449"
server_key_iv: "f0e826300623b9c1505d6c0b00525d3b"
server_key_version: "encryptionkey@0"
I20260926 20:46:30.267903 15720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:30.285912 15720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:30.288028 15720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:30.288210 15720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:30.291910 15720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:38437
I20260926 20:46:30.291952 15787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:38437 every 8 connection(s)
I20260926 20:46:30.292312 15720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:46:30.293241 15788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:30.295439 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 15720
I20260926 20:46:30.295567 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:46:30.295544 15788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: Bootstrap starting.
I20260926 20:46:30.295820 393 external_mini_cluster.cc:1428] Setting key b4171a787699b602c836ae2be44fbe63
I20260926 20:46:30.296355 15788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:30.296708 15788 log.cc:824] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:30.297482 15788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: No bootstrap required, opened a new log
I20260926 20:46:30.299921 15788 raft_consensus.cc:348] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60fabb67dd24bb59ec33c8673857818" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38437 } }
I20260926 20:46:30.300051 15788 raft_consensus.cc:374] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:30.300073 15788 raft_consensus.cc:729] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60fabb67dd24bb59ec33c8673857818, State: Initialized, Role: FOLLOWER
I20260926 20:46:30.300220 15788 consensus_queue.cc:260] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60fabb67dd24bb59ec33c8673857818" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38437 } }
I20260926 20:46:30.300299 15788 raft_consensus.cc:388] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:46:30.300340 15788 raft_consensus.cc:482] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:46:30.300390 15788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:30.301083 15788 raft_consensus.cc:504] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60fabb67dd24bb59ec33c8673857818" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38437 } }
I20260926 20:46:30.301205 15788 leader_election.cc:302] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a60fabb67dd24bb59ec33c8673857818; no voters:
I20260926 20:46:30.301447 15788 leader_election.cc:288] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:46:30.301491 15793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:46:30.301652 15793 raft_consensus.cc:686] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 1 LEADER]: Becoming Leader. State: Replica: a60fabb67dd24bb59ec33c8673857818, State: Running, Role: LEADER
I20260926 20:46:30.301795 15788 sys_catalog.cc:564] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:46:30.301795 15793 consensus_queue.cc:237] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60fabb67dd24bb59ec33c8673857818" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38437 } }
I20260926 20:46:30.302205 15795 sys_catalog.cc:455] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a60fabb67dd24bb59ec33c8673857818. Latest consensus state: current_term: 1 leader_uuid: "a60fabb67dd24bb59ec33c8673857818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60fabb67dd24bb59ec33c8673857818" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38437 } } }
I20260926 20:46:30.302304 15795 sys_catalog.cc:458] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [sys.catalog]: This master's current role is: LEADER
Sep 26 20:46:30 dist-test-slave-4hd3 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:30.303424 15794 sys_catalog.cc:455] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a60fabb67dd24bb59ec33c8673857818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60fabb67dd24bb59ec33c8673857818" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38437 } } }
W20260926 20:46:30.303821 15809 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 20:46:30.303861 15794 sys_catalog.cc:458] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [sys.catalog]: This master's current role is: LEADER
W20260926 20:46:30.303891 15809 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:46:30.303990 15800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:46:30.304162 15800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:46:30.305759 15800 catalog_manager.cc:1347] Generated new cluster ID: 7a1d8ca797b44af5a4ab2a43e4c41a1e
I20260926 20:46:30.305816 15800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:46:30.307936 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:30.296383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38010 (local address 127.0.98.126:38437)
0926 20:46:30.296566 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:30.296575 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:30.296609 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:46:30.297154 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:30.297164 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:30.297229 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:30.297380 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:30.298062 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.299509 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.300516 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.301132 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.301245 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:30.304204 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:30.304208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:30.304208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:30.304221 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:30.306406 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:30.306780 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:30.306783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:30.306783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:30.306869 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:30.307084 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:30.307084 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:30.307085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:30.307313 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:30.307474 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:30.307579 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 20:46:30.307745 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260926 20:46:30.315048 15800 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:46:30.315572 15800 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:46:30.319201 15800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: Generated new TSK 0
I20260926 20:46:30.319391 15800 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260926 20:46:30.347150 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38437
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:46:30.426607 15814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:30.426841 15814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:30.426877 15814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:30.428030 15814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:30.428097 15814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:30.428143 15814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:46:30.429561 15814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:30.429821 15814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:30.430047 15814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:30.434998 15819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:30.435038 15814 server_base.cc:1034] running on GCE node
W20260926 20:46:30.435015 15820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:30.434998 15822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:30.435433 15814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:30.435696 15814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:30.436892 15814 hybrid_clock.cc:648] HybridClock initialized: now 1790455590436881 us; error 31 us; skew 500 ppm
Sep 26 20:46:30 dist-test-slave-4hd3 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:30.439316 15814 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:46:30.439693 15814 webserver.cc:460] Webserver started at http://127.0.98.65:40231/ using document root <none> and password file <none>
I20260926 20:46:30.439929 15814 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:30.439971 15814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:30.440059 15814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:46:30.441056 15814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "d9243cda34234a07abb66bcb8a40d1a1"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "25ea6f88a4842d03bbd8d8f9b9fe5023"
server_key_iv: "135b890d119c649729ff28de8cd64839"
server_key_version: "encryptionkey@0"
I20260926 20:46:30.441358 15814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "d9243cda34234a07abb66bcb8a40d1a1"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "25ea6f88a4842d03bbd8d8f9b9fe5023"
server_key_iv: "135b890d119c649729ff28de8cd64839"
server_key_version: "encryptionkey@0"
I20260926 20:46:30.442592 15814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:46:30.443302 15829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:30.443511 15814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:46:30.443581 15814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "d9243cda34234a07abb66bcb8a40d1a1"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "25ea6f88a4842d03bbd8d8f9b9fe5023"
server_key_iv: "135b890d119c649729ff28de8cd64839"
server_key_version: "encryptionkey@0"
I20260926 20:46:30.443634 15814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:30.452105 15814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:30.453838 15814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:30.453948 15814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:30.454123 15814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:30.454365 15814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:46:30.454394 15814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:30.454414 15814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:46:30.454427 15814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:30.460090 15814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:38855
I20260926 20:46:30.460186 15942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:38855 every 8 connection(s)
I20260926 20:46:30.460479 15814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:46:30.462117 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 15814
I20260926 20:46:30.462247 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:46:30.462450 393 external_mini_cluster.cc:1428] Setting key 0fc045a28eae072991f2f2d393d47a09
Sep 26 20:46:30 dist-test-slave-4hd3 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260926 20:46:30.471339 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:30.461193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:39943 (local address 127.0.98.126:38437)
0926 20:46:30.461252 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:30.461255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:30.461464 (+ 209us) server_negotiation.cc:407] Connection header received
0926 20:46:30.462041 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:30.462043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:30.462057 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:30.462124 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:30.463263 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.463857 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.465306 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.465504 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.465568 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:30.468033 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:30.468035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:30.468035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:30.468044 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:30.469772 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:30.470240 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:30.470242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:30.470242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:30.470328 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:30.470582 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:30.470583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:30.470584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:30.470734 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:30.470816 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:30.471060 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 20:46:30.471164 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 20:46:30.471417 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:30.461179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:39943)
0926 20:46:30.461342 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:30.461377 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:30.461893 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:30.462198 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:30.462203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:30.462928 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:30.463179 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:30.463185 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.463948 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.463950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:30.465213 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:30.465216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.465588 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.465590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:30.465743 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:30.466257 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:30.466268 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:30.467942 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:30.469871 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:30.469878 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:30.469880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:30.470145 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:30.470402 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:30.470403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:30.470403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:30.470503 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:30.470809 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:30.470814 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:30.470929 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 20:46:30.471016 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 20:46:30.471154 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260926 20:46:30.471977 15943 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38437
I20260926 20:46:30.472115 15943 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:30.472316 15943 heartbeater.cc:511] Master 127.0.98.126:38437 requested a full tablet report, sending...
I20260926 20:46:30.472831 15752 ts_manager.cc:194] Registered new tserver with Master: d9243cda34234a07abb66bcb8a40d1a1 (127.0.98.65:38855)
I20260926 20:46:30.473538 15752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:39943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260926 20:46:30.488030 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38437
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:46:30.568426 15948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:30.568691 15948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:30.568728 15948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:30.569900 15948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:30.569971 15948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:30.570016 15948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:46:30.571544 15948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:30.571854 15948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:30.572086 15948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:46:30.577093 15948 server_base.cc:1034] running on GCE node
W20260926 20:46:30.577018 15953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:30.577020 15956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:30.577035 15954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:30.577421 15948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:30.577687 15948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:30.578904 15948 hybrid_clock.cc:648] HybridClock initialized: now 1790455590578879 us; error 34 us; skew 500 ppm
Sep 26 20:46:30 dist-test-slave-4hd3 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:30.581568 15948 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:46:30.582077 15948 webserver.cc:460] Webserver started at http://127.0.98.66:40745/ using document root <none> and password file <none>
I20260926 20:46:30.582319 15948 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:30.582384 15948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:30.582535 15948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:46:30.583575 15948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "5e09a8ec28794781968b814593e4c8bf"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "aceda51da202173d2abbaef6bd355c9c"
server_key_iv: "dafc86ba874fe5d7853d78e98f64a5f0"
server_key_version: "encryptionkey@0"
I20260926 20:46:30.583961 15948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "5e09a8ec28794781968b814593e4c8bf"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "aceda51da202173d2abbaef6bd355c9c"
server_key_iv: "dafc86ba874fe5d7853d78e98f64a5f0"
server_key_version: "encryptionkey@0"
I20260926 20:46:30.585278 15948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:46:30.586033 15963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:30.586208 15948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:46:30.586302 15948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "5e09a8ec28794781968b814593e4c8bf"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "aceda51da202173d2abbaef6bd355c9c"
server_key_iv: "dafc86ba874fe5d7853d78e98f64a5f0"
server_key_version: "encryptionkey@0"
I20260926 20:46:30.586382 15948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:30.605374 15948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:30.607339 15948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:30.607503 15948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:30.607872 15948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:30.608201 15948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:46:30.608254 15948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:30.608297 15948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:46:30.608325 15948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:30.614032 15948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:39345
I20260926 20:46:30.614073 16076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:39345 every 8 connection(s)
I20260926 20:46:30.614423 15948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:46:30 dist-test-slave-4hd3 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:30.622658 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 15948
I20260926 20:46:30.622781 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:46:30.623039 393 external_mini_cluster.cc:1428] Setting key 86c78f3788283d17009184dc971f76b6
I20260926 20:46:30.623970 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:30.615203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:44827 (local address 127.0.98.126:38437)
0926 20:46:30.615285 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:30.615287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:30.615491 (+ 204us) server_negotiation.cc:407] Connection header received
0926 20:46:30.615984 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:30.615985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:30.615995 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:30.616057 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:30.616958 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.617448 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.618459 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.618646 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.618700 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:30.620797 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:30.620798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:30.620799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:30.620804 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:30.622311 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:30.622841 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:30.622842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:30.622842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:30.622910 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:30.623228 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:30.623229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:30.623230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:30.623384 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:30.623474 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:30.623818 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 20:46:30.623902 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:46:30.624267 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:30.615187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:44827)
0926 20:46:30.615354 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:30.615396 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:30.615895 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:30.616114 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:30.616118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:30.616685 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:30.616875 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:30.616880 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.617535 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.617536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:30.618376 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:30.618379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.618720 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.618720 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:30.618816 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:30.619330 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:30.619338 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:30.620698 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:30.622406 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:30.622414 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:30.622417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:30.622723 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:30.622982 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:30.622983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:30.622984 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:30.623134 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:30.623488 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:30.623493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:30.623625 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 20:46:30.623794 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 20:46:30.623978 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260926 20:46:30.624722 16077 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38437
I20260926 20:46:30.624827 16077 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:30.624995 16077 heartbeater.cc:511] Master 127.0.98.126:38437 requested a full tablet report, sending...
I20260926 20:46:30.625385 15752 ts_manager.cc:194] Registered new tserver with Master: 5e09a8ec28794781968b814593e4c8bf (127.0.98.66:39345)
I20260926 20:46:30.625783 15752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:44827
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260926 20:46:30.650247 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38437
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:46:30.731109 16082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:30.731331 16082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:30.731365 16082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:30.732556 16082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:30.732609 16082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:30.732636 16082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:46:30.734041 16082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:30.734239 16082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:30.734493 16082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:30.739444 16090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:30.739503 16087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:30.739466 16088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:30.739692 16082 server_base.cc:1034] running on GCE node
I20260926 20:46:30.739917 16082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:30.740173 16082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:30.741349 16082 hybrid_clock.cc:648] HybridClock initialized: now 1790455590741339 us; error 30 us; skew 500 ppm
Sep 26 20:46:30 dist-test-slave-4hd3 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:30.743852 16082 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:46:30.744325 16082 webserver.cc:460] Webserver started at http://127.0.98.67:46553/ using document root <none> and password file <none>
I20260926 20:46:30.744577 16082 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:30.744654 16082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:30.744791 16082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:46:30.745837 16082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "28bc76dae6da4a66b0bbe69e485e27f2"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "77ee7967a212333bf9625e1706e4b83c"
server_key_iv: "5b3665c09bd889f3d3d52eec5a86d287"
server_key_version: "encryptionkey@0"
I20260926 20:46:30.746208 16082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "28bc76dae6da4a66b0bbe69e485e27f2"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "77ee7967a212333bf9625e1706e4b83c"
server_key_iv: "5b3665c09bd889f3d3d52eec5a86d287"
server_key_version: "encryptionkey@0"
I20260926 20:46:30.747612 16082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:46:30.748446 16097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:30.748781 16082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:46:30.748894 16082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "28bc76dae6da4a66b0bbe69e485e27f2"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "77ee7967a212333bf9625e1706e4b83c"
server_key_iv: "5b3665c09bd889f3d3d52eec5a86d287"
server_key_version: "encryptionkey@0"
I20260926 20:46:30.748965 16082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:30.765748 16082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:30.768177 16082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:30.768342 16082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:30.768567 16082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:30.768880 16082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:46:30.768913 16082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:30.768953 16082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:46:30.768980 16082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:30.774923 16082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:42389
I20260926 20:46:30.774966 16210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:42389 every 8 connection(s)
I20260926 20:46:30.775295 16082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:46:30 dist-test-slave-4hd3 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:30.784564 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:30.776107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:39035 (local address 127.0.98.126:38437)
0926 20:46:30.776167 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:30.776169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:30.776371 (+ 202us) server_negotiation.cc:407] Connection header received
0926 20:46:30.776866 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:30.776867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:30.776877 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:30.776938 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:30.777907 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.778424 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.779456 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.779653 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.779798 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:30.781983 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:30.781984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:30.781984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:30.781991 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:30.783257 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:30.783679 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:30.783680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:30.783680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:30.783785 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:30.784028 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:30.784028 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:30.784029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:30.784147 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:30.784200 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:30.784440 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 20:46:30.784499 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:46:30.784780 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:30.776066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:39035)
0926 20:46:30.776242 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:30.776285 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:30.776768 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:30.777008 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:30.777011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:30.777616 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:30.777817 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:30.777821 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.778529 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.778530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:30.779344 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:30.779346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.779824 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.779825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:30.779946 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:30.780464 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:30.780473 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:30.781859 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:30.783335 (+ 1476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:30.783357 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:30.783358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:30.783607 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:30.783853 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:30.783854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:30.783854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:30.783961 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:30.784205 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:30.784209 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:30.784326 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 20:46:30.784410 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:46:30.784529 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":111,"threads_started":1}
I20260926 20:46:30.785239 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 16082
I20260926 20:46:30.785349 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:46:30.785360 16211 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38437
I20260926 20:46:30.785444 16211 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:30.785552 393 external_mini_cluster.cc:1428] Setting key 5dc4534d88381911d348743d2cce9216
I20260926 20:46:30.785638 16211 heartbeater.cc:511] Master 127.0.98.126:38437 requested a full tablet report, sending...
I20260926 20:46:30.785989 15752 ts_manager.cc:194] Registered new tserver with Master: 28bc76dae6da4a66b0bbe69e485e27f2 (127.0.98.67:42389)
I20260926 20:46:30.786427 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:46:30.786461 15752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:39035
I20260926 20:46:30.793819 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:30.787834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38024 (local address 127.0.98.126:38437)
0926 20:46:30.787899 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:30.787901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:30.788085 (+ 184us) server_negotiation.cc:407] Connection header received
0926 20:46:30.788118 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:30.788118 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:30.788126 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:30.788164 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:30.788690 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.789163 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.790041 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.790233 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.790296 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:30.791481 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:30.791482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:30.791483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:30.791489 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:30.792754 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:30.793074 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:30.793075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:30.793075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:30.793129 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:30.793281 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:30.793282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:30.793282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:30.793402 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:30.793472 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:30.793619 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 20:46:30.793680 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:46:30.795607 15752 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:46:30.797272 15752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:46:30.803624 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:30.799359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39345 (local address 127.0.0.1:60696)
0926 20:46:30.799532 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:30.799571 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:30.799636 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:30.799886 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:30.799889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:30.799892 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:30.799958 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:30.799960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.800993 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.800994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:30.802114 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:30.802116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.803165 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.803166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:30.803254 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:30.803268 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 20:46:30.803495 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 20:46:30.803552 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":29,"thread_start_us":65,"threads_started":1}
I20260926 20:46:30.803642 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:30.799445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60696 (local address 127.0.98.66:39345)
0926 20:46:30.799620 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:30.799625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:30.799637 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:46:30.799758 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:30.799760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:30.799776 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:30.799868 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:30.800044 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.800881 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.802209 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.803039 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.803135 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:30.803280 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:30.803518 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 20:46:30.803568 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260926 20:46:30.803931 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:30.799569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:38855 (local address 127.0.0.1:54964)
0926 20:46:30.799983 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:30.799993 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:30.800041 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:30.800386 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:30.800387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:30.800389 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:30.800420 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:30.800421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.801711 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.801712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:30.802634 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:30.802637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.803667 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.803668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:30.803762 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:30.803768 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:46:30.803825 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 20:46:30.803867 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":55,"threads_started":1}
I20260926 20:46:30.804215 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:30.799811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54964 (local address 127.0.98.65:38855)
0926 20:46:30.800178 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:30.800183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:30.800192 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:46:30.800229 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:30.800231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:30.800244 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:30.800312 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:30.800495 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.801605 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.802724 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.803490 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.803560 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:30.803630 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:30.804077 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 20:46:30.804119 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":81,"threads_started":1}
I20260926 20:46:30.804304 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:30.799366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:42389 (local address 127.0.0.1:38612)
0926 20:46:30.799910 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:30.799919 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:30.799979 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:30.800291 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:30.800292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:30.800294 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:30.800342 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:30.800343 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.801605 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.801606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:30.802803 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:30.802806 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.804067 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.804068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:30.804135 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:30.804138 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:46:30.804187 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 20:46:30.804223 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":38,"thread_start_us":69,"threads_started":1}
I20260926 20:46:30.804382 16011 tablet_service.cc:1467] Processing CreateTablet for tablet 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 (DEFAULT_TABLE table=test-table [id=0b317abdcee547c7a1fc94a5f61a2d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:46:30.804627 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:30.799657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.0.98.67:42389)
0926 20:46:30.799868 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:30.799872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:30.799979 (+ 107us) server_negotiation.cc:407] Connection header received
0926 20:46:30.800048 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:30.800050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:30.800066 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:30.800147 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:30.800420 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.801507 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.802906 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.803580 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.803639 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:30.804030 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:30.804220 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 20:46:30.804492 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":98,"threads_started":1}
I20260926 20:46:30.804742 16011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d2b9d6eb3c442a3ab1ba0ee64a03bd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:30.804834 15877 tablet_service.cc:1467] Processing CreateTablet for tablet 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 (DEFAULT_TABLE table=test-table [id=0b317abdcee547c7a1fc94a5f61a2d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:46:30.805176 15877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d2b9d6eb3c442a3ab1ba0ee64a03bd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:30.805366 16145 tablet_service.cc:1467] Processing CreateTablet for tablet 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 (DEFAULT_TABLE table=test-table [id=0b317abdcee547c7a1fc94a5f61a2d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:46:30.805647 16145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d2b9d6eb3c442a3ab1ba0ee64a03bd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:30.807458 16225 tablet_bootstrap.cc:492] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf: Bootstrap starting.
I20260926 20:46:30.808151 16225 tablet_bootstrap.cc:654] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:30.808483 16226 tablet_bootstrap.cc:492] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1: Bootstrap starting.
I20260926 20:46:30.808723 16225 log.cc:824] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:30.808804 16227 tablet_bootstrap.cc:492] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2: Bootstrap starting.
I20260926 20:46:30.809137 16226 tablet_bootstrap.cc:654] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:30.809427 16227 tablet_bootstrap.cc:654] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:30.809652 16226 log.cc:824] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:30.809775 16225 tablet_bootstrap.cc:492] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf: No bootstrap required, opened a new log
I20260926 20:46:30.809850 16225 ts_tablet_manager.cc:1397] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:46:30.809978 16227 log.cc:824] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:30.810380 16226 tablet_bootstrap.cc:492] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1: No bootstrap required, opened a new log
I20260926 20:46:30.810453 16226 ts_tablet_manager.cc:1397] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:46:30.810688 16227 tablet_bootstrap.cc:492] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2: No bootstrap required, opened a new log
I20260926 20:46:30.810775 16227 ts_tablet_manager.cc:1397] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:46:30.811921 16225 raft_consensus.cc:348] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:30.812108 16225 raft_consensus.cc:374] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:30.812157 16225 raft_consensus.cc:729] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e09a8ec28794781968b814593e4c8bf, State: Initialized, Role: FOLLOWER
I20260926 20:46:30.812296 16225 consensus_queue.cc:260] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:30.812441 16226 raft_consensus.cc:348] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:30.812580 16225 ts_tablet_manager.cc:1428] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:46:30.812605 16226 raft_consensus.cc:374] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:30.812510 16227 raft_consensus.cc:348] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:30.812640 16226 raft_consensus.cc:729] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9243cda34234a07abb66bcb8a40d1a1, State: Initialized, Role: FOLLOWER
I20260926 20:46:30.812690 16227 raft_consensus.cc:374] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:30.812736 16227 raft_consensus.cc:729] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28bc76dae6da4a66b0bbe69e485e27f2, State: Initialized, Role: FOLLOWER
I20260926 20:46:30.812785 16226 consensus_queue.cc:260] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:30.812851 16227 consensus_queue.cc:260] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:30.813097 16226 ts_tablet_manager.cc:1428] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:46:30.813136 16227 ts_tablet_manager.cc:1428] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:46:30.813212 16077 heartbeater.cc:503] Master 127.0.98.126:38437 was elected leader, sending a full tablet report...
I20260926 20:46:30.813439 16211 heartbeater.cc:503] Master 127.0.98.126:38437 was elected leader, sending a full tablet report...
I20260926 20:46:30.813097 15943 heartbeater.cc:503] Master 127.0.98.126:38437 was elected leader, sending a full tablet report...
W20260926 20:46:30.864990 16078 tablet.cc:2369] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:46:30.961228 15944 tablet.cc:2369] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:46:30.988778 16233 raft_consensus.cc:482] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:46:30.988924 16233 raft_consensus.cc:504] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:30.989277 16233 leader_election.cc:288] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e09a8ec28794781968b814593e4c8bf (127.0.98.66:39345), d9243cda34234a07abb66bcb8a40d1a1 (127.0.98.65:38855)
I20260926 20:46:30.992213 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:30.989718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:47631 (local address 127.0.98.65:38855)
0926 20:46:30.989766 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:30.989768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:30.989778 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:46:30.989836 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:30.989836 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:30.989848 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:30.989902 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:30.990026 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.990486 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.991455 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.991866 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.991948 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:30.991958 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:30.992108 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 20:46:30.992144 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 20:46:30.992237 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:30.989594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:37577 (local address 127.0.98.66:39345)
0926 20:46:30.989683 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:30.989686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:30.989695 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:46:30.989724 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:30.989725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:30.989737 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:30.989825 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:30.989991 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.990481 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.991430 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.991874 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.991957 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:30.991967 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:30.992136 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 20:46:30.992172 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:46:30.992237 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:30.989469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39345 (local address 127.0.98.67:37577)
0926 20:46:30.989528 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:30.989540 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:30.989619 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:30.989830 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:30.989831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:30.989833 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:30.989905 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:30.989907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.990569 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.990570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:30.991336 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:30.991338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.991958 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.991959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:30.992048 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:30.992051 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:46:30.992110 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 20:46:30.992153 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":1}
I20260926 20:46:30.992241 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:30.989579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:38855 (local address 127.0.98.67:47631)
0926 20:46:30.989722 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:30.989730 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:30.989768 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:30.989904 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:30.989904 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:30.989905 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:30.989961 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:30.989963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.990569 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.990570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:30.991365 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:30.991367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:30.991949 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:30.991950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:30.992029 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:30.992033 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:46:30.992102 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:46:30.992141 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":48,"thread_start_us":86,"threads_started":1}
I20260926 20:46:30.992678 15897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2b9d6eb3c442a3ab1ba0ee64a03bd8" candidate_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9243cda34234a07abb66bcb8a40d1a1" is_pre_election: true
I20260926 20:46:30.992847 15897 raft_consensus.cc:2454] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28bc76dae6da4a66b0bbe69e485e27f2 in term 0.
I20260926 20:46:30.993100 16098 leader_election.cc:302] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28bc76dae6da4a66b0bbe69e485e27f2, d9243cda34234a07abb66bcb8a40d1a1; no voters:
I20260926 20:46:30.993072 16031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2b9d6eb3c442a3ab1ba0ee64a03bd8" candidate_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e09a8ec28794781968b814593e4c8bf" is_pre_election: true
I20260926 20:46:30.993209 16031 raft_consensus.cc:2454] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28bc76dae6da4a66b0bbe69e485e27f2 in term 0.
I20260926 20:46:30.993331 16233 raft_consensus.cc:2781] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:46:30.993405 16233 raft_consensus.cc:482] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:46:30.993440 16233 raft_consensus.cc:3037] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:30.994158 16233 raft_consensus.cc:504] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:30.994294 16233 leader_election.cc:288] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [CANDIDATE]: Term 1 election: Requested vote from peers 5e09a8ec28794781968b814593e4c8bf (127.0.98.66:39345), d9243cda34234a07abb66bcb8a40d1a1 (127.0.98.65:38855)
I20260926 20:46:30.994566 15897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2b9d6eb3c442a3ab1ba0ee64a03bd8" candidate_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9243cda34234a07abb66bcb8a40d1a1"
I20260926 20:46:30.994549 16031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2b9d6eb3c442a3ab1ba0ee64a03bd8" candidate_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e09a8ec28794781968b814593e4c8bf"
I20260926 20:46:30.994643 15897 raft_consensus.cc:3037] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:30.994675 16031 raft_consensus.cc:3037] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:30.995558 15897 raft_consensus.cc:2454] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28bc76dae6da4a66b0bbe69e485e27f2 in term 1.
I20260926 20:46:30.995558 16031 raft_consensus.cc:2454] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28bc76dae6da4a66b0bbe69e485e27f2 in term 1.
I20260926 20:46:30.995777 16098 leader_election.cc:302] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28bc76dae6da4a66b0bbe69e485e27f2, 5e09a8ec28794781968b814593e4c8bf; no voters:
I20260926 20:46:30.995883 16233 raft_consensus.cc:2781] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:46:30.996031 16233 raft_consensus.cc:686] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 1 LEADER]: Becoming Leader. State: Replica: 28bc76dae6da4a66b0bbe69e485e27f2, State: Running, Role: LEADER
I20260926 20:46:30.996148 16233 consensus_queue.cc:237] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:30.997025 15751 catalog_manager.cc:5495] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28bc76dae6da4a66b0bbe69e485e27f2 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } health_report { overall_health: UNKNOWN } } }
W20260926 20:46:31.025903 16212 tablet.cc:2369] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:46:31.030534 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:31.028238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.0.98.67:42389)
0926 20:46:31.028306 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:31.028308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:31.028320 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:46:31.028367 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:31.028367 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:31.028385 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:31.028444 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:46:31.028572 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:31.029023 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:31.029758 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:31.029940 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:31.029995 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:31.030178 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:46:31.030275 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:46:31.030349 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:31.030382 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 20:46:31.030400 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:46:31.032423 15897 raft_consensus.cc:1261] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Refusing update from remote peer 28bc76dae6da4a66b0bbe69e485e27f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:46:31.032428 16031 raft_consensus.cc:1261] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf [term 1 FOLLOWER]: Refusing update from remote peer 28bc76dae6da4a66b0bbe69e485e27f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:46:31.032820 16233 consensus_queue.cc:1035] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:46:31.032995 16235 consensus_queue.cc:1035] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:46:31.035645 16239 mvcc.cc:205] Tried to move back new op lower bound from 7333706100865744896 to 7333706100720922624. Current Snapshot: MvccSnapshot[applied={T|T < 7333706100865744896}]
I20260926 20:46:31.037683 15752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:46:31.037781 15752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:46:31.038532 15752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: Sending DeleteTablet for 3 replicas of tablet 2d2b9d6eb3c442a3ab1ba0ee64a03bd8
I20260926 20:46:31.038869 16011 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:46:31 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:60696
I20260926 20:46:31.038937 15877 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:46:31 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:54964
I20260926 20:46:31.039067 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 15814
I20260926 20:46:31.043854 16145 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:46:31 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:38612
I20260926 20:46:31.043943 16248 tablet_replica.cc:330] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf: stopping tablet replica
I20260926 20:46:31.044085 16248 raft_consensus.cc:2229] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:46:31.044101 16250 tablet_replica.cc:330] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2: stopping tablet replica
I20260926 20:46:31.044198 16250 raft_consensus.cc:2229] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:46:31.044216 16248 raft_consensus.cc:2258] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:46:31.044389 16250 raft_consensus.cc:2258] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:46:31.044703 16248 ts_tablet_manager.cc:1905] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:46:31.044868 16250 ts_tablet_manager.cc:1905] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:46:31.044854 15736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:38855 (error 108)
I20260926 20:46:31.045504 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 15948
I20260926 20:46:31.047415 16250 ts_tablet_manager.cc:1918] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:46:31.047501 16250 log.cc:1196] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/2d2b9d6eb3c442a3ab1ba0ee64a03bd8
I20260926 20:46:31.047796 16250 ts_tablet_manager.cc:1939] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 28bc76dae6da4a66b0bbe69e485e27f2: Deleting consensus metadata
I20260926 20:46:31.048250 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:31.048113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:38855 (local address 127.0.0.1:54966)
0926 20:46:31.048166 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:31.048207 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 20:46:31.048389 15736 catalog_manager.cc:4584] TS d9243cda34234a07abb66bcb8a40d1a1 (127.0.98.65:38855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d2b9d6eb3c442a3ab1ba0ee64a03bd8: Network error: Client connection negotiation failed: client connection to 127.0.98.65:38855: connect: Connection refused (error 111)
I20260926 20:46:31.048416 15739 catalog_manager.cc:4841] TS 28bc76dae6da4a66b0bbe69e485e27f2 (127.0.98.67:42389): tablet 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 (table test-table [id=0b317abdcee547c7a1fc94a5f61a2d1d]) successfully deleted
I20260926 20:46:31.052681 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 16082
I20260926 20:46:31.052732 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:31.052627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39345 (local address 127.0.0.1:60706)
0926 20:46:31.052669 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:31.052703 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.66:39345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260926 20:46:31.057713 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 15720
I20260926 20:46:51.063558 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38437
--webserver_interface=127.0.98.126
--webserver_port=40097
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:38437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:46:51.145553 16252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:51.145809 16252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:51.145845 16252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:51.147025 16252 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:46:51.147096 16252 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:46:51.147118 16252 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:46:51.147136 16252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:46:51.147167 16252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:51.147187 16252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:51.147212 16252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:46:51.147236 16252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:46:51.148723 16252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:38437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=40097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:51.149016 16252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:51.149247 16252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:51.158097 16258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:51.158174 16261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:51.158174 16259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:51.158175 16252 server_base.cc:1034] running on GCE node
I20260926 20:46:51.158639 16252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:51.158967 16252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:51.160133 16252 hybrid_clock.cc:648] HybridClock initialized: now 1790455611160119 us; error 48 us; skew 500 ppm
Sep 26 20:46:51 dist-test-slave-4hd3 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:51.162634 16252 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:46:51.163190 16252 webserver.cc:460] Webserver started at http://127.0.98.126:40097/ using document root <none> and password file <none>
I20260926 20:46:51.163444 16252 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:51.163532 16252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:51.165040 16252 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:46:51.165797 16268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:51.166006 16252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:46:51.166117 16252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "a60fabb67dd24bb59ec33c8673857818"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "9e3d30525cb39c28e21c8401ce659449"
server_key_iv: "f0e826300623b9c1505d6c0b00525d3b"
server_key_version: "encryptionkey@0"
I20260926 20:46:51.166538 16252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:51.189981 16252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:51.192435 16252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:51.192606 16252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:51.196684 16252 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:38437
I20260926 20:46:51.196751 16320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:38437 every 8 connection(s)
I20260926 20:46:51.197155 16252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:46:51.198700 16321 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:46:51.198998 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 16252
I20260926 20:46:51.199277 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:38855
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:38437
--webserver_port=40231
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:46:51.199515 16321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: Bootstrap starting.
I20260926 20:46:51.210783 16321 log.cc:824] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:51.213243 16321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:46:51.213563 16321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: Bootstrap complete.
I20260926 20:46:51.216749 16321 raft_consensus.cc:348] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60fabb67dd24bb59ec33c8673857818" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38437 } }
I20260926 20:46:51.217054 16321 raft_consensus.cc:729] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60fabb67dd24bb59ec33c8673857818, State: Initialized, Role: FOLLOWER
I20260926 20:46:51.217211 16321 consensus_queue.cc:260] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60fabb67dd24bb59ec33c8673857818" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38437 } }
I20260926 20:46:51.217278 16321 raft_consensus.cc:388] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:46:51.217319 16321 raft_consensus.cc:482] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:46:51.217365 16321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:46:51.218699 16321 raft_consensus.cc:504] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60fabb67dd24bb59ec33c8673857818" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38437 } }
I20260926 20:46:51.218812 16321 leader_election.cc:302] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a60fabb67dd24bb59ec33c8673857818; no voters:
I20260926 20:46:51.218986 16321 leader_election.cc:288] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:46:51.219034 16325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:46:51.219223 16325 raft_consensus.cc:686] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 2 LEADER]: Becoming Leader. State: Replica: a60fabb67dd24bb59ec33c8673857818, State: Running, Role: LEADER
I20260926 20:46:51.219264 16321 sys_catalog.cc:564] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:46:51.219323 16325 consensus_queue.cc:237] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60fabb67dd24bb59ec33c8673857818" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38437 } }
I20260926 20:46:51.219683 16326 sys_catalog.cc:455] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a60fabb67dd24bb59ec33c8673857818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60fabb67dd24bb59ec33c8673857818" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38437 } } }
I20260926 20:46:51.219794 16326 sys_catalog.cc:458] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [sys.catalog]: This master's current role is: LEADER
I20260926 20:46:51.220009 16327 sys_catalog.cc:455] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a60fabb67dd24bb59ec33c8673857818. Latest consensus state: current_term: 2 leader_uuid: "a60fabb67dd24bb59ec33c8673857818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60fabb67dd24bb59ec33c8673857818" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38437 } } }
I20260926 20:46:51.220121 16332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:46:51.220132 16327 sys_catalog.cc:458] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [sys.catalog]: This master's current role is: LEADER
I20260926 20:46:51.221011 16332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:46:51.221113 16332 catalog_manager.cc:1259] Loaded cluster ID: 7a1d8ca797b44af5a4ab2a43e4c41a1e
I20260926 20:46:51.221143 16332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:46:51.222458 16332 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:46:51.222635 16332 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: Loaded TSK: 0
I20260926 20:46:51.238631 16332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: Generated new TSK 1
I20260926 20:46:51.238749 16332 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:46:51.290892 16323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:51.291126 16323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:51.291167 16323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:51.292402 16323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:51.292472 16323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:51.292518 16323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:46:51.293967 16323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:38855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=40231
--tserver_master_addrs=127.0.98.126:38437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:51.294229 16323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:51.294523 16323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:51.300542 16349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:51.300539 16351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:51.300554 16348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:51.300781 16323 server_base.cc:1034] running on GCE node
I20260926 20:46:51.301179 16323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:51.301453 16323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:51.302680 16323 hybrid_clock.cc:648] HybridClock initialized: now 1790455611302661 us; error 31 us; skew 500 ppm
Sep 26 20:46:51 dist-test-slave-4hd3 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:51.305717 16323 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:46:51.306279 16323 webserver.cc:460] Webserver started at http://127.0.98.65:40231/ using document root <none> and password file <none>
I20260926 20:46:51.306541 16323 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:51.306638 16323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:51.308204 16323 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:46:51.308979 16358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:51.309208 16323 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:46:51.309311 16323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "d9243cda34234a07abb66bcb8a40d1a1"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "25ea6f88a4842d03bbd8d8f9b9fe5023"
server_key_iv: "135b890d119c649729ff28de8cd64839"
server_key_version: "encryptionkey@0"
I20260926 20:46:51.309661 16323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:51.319070 16323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:51.321420 16323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:51.321578 16323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:51.321797 16323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:51.322211 16365 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:46:51.324126 16323 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:46:51.324205 16323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 20:46:51.324254 16323 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:46:51.325018 16323 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:46:51.325079 16323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 20:46:51.325146 16365 tablet_bootstrap.cc:492] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1: Bootstrap starting.
I20260926 20:46:51.333165 16323 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:38855
I20260926 20:46:51.333206 16472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:38855 every 8 connection(s)
I20260926 20:46:51.333612 16323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:46:51.335088 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 16323
I20260926 20:46:51.335453 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:39345
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:38437
--webserver_port=40745
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:46:51.336119 16365 log.cc:824] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:51.338909 16365 tablet_bootstrap.cc:492] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:46:51.339216 16365 tablet_bootstrap.cc:492] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1: Bootstrap complete.
I20260926 20:46:51.340449 16365 ts_tablet_manager.cc:1397] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260926 20:46:51.341830 16365 raft_consensus.cc:348] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:51.342154 16365 raft_consensus.cc:729] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9243cda34234a07abb66bcb8a40d1a1, State: Initialized, Role: FOLLOWER
I20260926 20:46:51.342315 16365 consensus_queue.cc:260] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:51.342625 16365 ts_tablet_manager.cc:1428] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
Sep 26 20:46:51 dist-test-slave-4hd3 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:51.348889 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.334392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:44739)
0926 20:46:51.336457 (+ 2065us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:51.336507 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:51.337133 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:51.338019 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:51.338024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:51.338391 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:51.338663 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:51.338669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.340750 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.340751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:51.342025 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:51.342027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.343120 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.343120 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:51.343209 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.343844 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:51.343852 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:51.345202 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:51.347173 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:51.347179 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:51.347182 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:51.347417 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:51.347686 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:51.347687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:51.347687 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:51.347840 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:51.348262 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:51.348267 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:51.348398 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 20:46:51.348502 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 20:46:51.348674 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2020,"thread_start_us":97,"threads_started":1}
I20260926 20:46:51.348881 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.334476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:44739 (local address 127.0.98.126:38437)
0926 20:46:51.337513 (+ 3037us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:51.337524 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:51.337567 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:46:51.337703 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:51.337709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:51.337761 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:51.337945 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:51.338818 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.340624 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.342130 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.342965 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.343026 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.345333 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:51.345336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:51.345337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:51.345348 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:51.347063 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:51.347521 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:51.347524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:51.347524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:51.347612 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:51.347928 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:51.347929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:51.347929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:51.348179 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:51.348282 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:51.348568 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 20:46:51.348690 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2986,"thread_start_us":79,"threads_started":1}
I20260926 20:46:51.350080 16473 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38437
I20260926 20:46:51.350172 16473 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:51.350360 16473 heartbeater.cc:511] Master 127.0.98.126:38437 requested a full tablet report, sending...
I20260926 20:46:51.351058 16285 ts_manager.cc:194] Registered new tserver with Master: d9243cda34234a07abb66bcb8a40d1a1 (127.0.98.65:38855)
I20260926 20:46:51.352180 16285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:44739
I20260926 20:46:51.353034 16473 heartbeater.cc:503] Master 127.0.98.126:38437 was elected leader, sending a full tablet report...
I20260926 20:46:51.358963 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.351641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:38855 (local address 127.0.0.1:39222)
0926 20:46:51.355834 (+ 4193us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:51.355879 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:51.355968 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:51.356169 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:51.356171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:51.356173 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:51.356257 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:51.356259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.357094 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.357095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:51.357949 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:51.357951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.358611 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.358611 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:51.358712 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.358724 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 20:46:51.358827 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 20:46:51.358883 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4163,"thread_start_us":115,"threads_started":1}
I20260926 20:46:51.358966 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.351907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.0.98.65:38855)
0926 20:46:51.352511 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:51.352515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:51.355951 (+ 3436us) server_negotiation.cc:407] Connection header received
0926 20:46:51.356044 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:51.356046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:51.356061 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:51.356161 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:51.356338 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.357005 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.358054 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.358521 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.358610 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.358665 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:51.358851 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 20:46:51.358888 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":70,"threads_started":1}
I20260926 20:46:51.359647 16390 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:46:31 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:39222
I20260926 20:46:51.360280 16482 tablet_replica.cc:330] stopping tablet replica
I20260926 20:46:51.360414 16482 raft_consensus.cc:2229] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:46:51.360483 16482 raft_consensus.cc:2258] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:46:51.360776 16482 ts_tablet_manager.cc:1905] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:46:51.363111 16482 ts_tablet_manager.cc:1918] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:46:51.363204 16482 log.cc:1196] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/wal/wals/2d2b9d6eb3c442a3ab1ba0ee64a03bd8
I20260926 20:46:51.363471 16482 ts_tablet_manager.cc:1939] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P d9243cda34234a07abb66bcb8a40d1a1: Deleting consensus metadata
I20260926 20:46:51.364032 16269 catalog_manager.cc:4841] TS d9243cda34234a07abb66bcb8a40d1a1 (127.0.98.65:38855): tablet 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 (table test-table [id=0b317abdcee547c7a1fc94a5f61a2d1d]) successfully deleted
W20260926 20:46:51.428830 16478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:51.429029 16478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:51.429052 16478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:51.430116 16478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:51.430152 16478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:51.430177 16478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:46:51.431514 16478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:39345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=40745
--tserver_master_addrs=127.0.98.126:38437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:51.431690 16478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:51.431995 16478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:51.437007 16490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:51.437021 16488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:51.437000 16487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:51.437040 16478 server_base.cc:1034] running on GCE node
I20260926 20:46:51.437392 16478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:51.437669 16478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:51.438848 16478 hybrid_clock.cc:648] HybridClock initialized: now 1790455611438835 us; error 29 us; skew 500 ppm
Sep 26 20:46:51 dist-test-slave-4hd3 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:51.441413 16478 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:46:51.441820 16478 webserver.cc:460] Webserver started at http://127.0.98.66:40745/ using document root <none> and password file <none>
I20260926 20:46:51.442063 16478 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:51.442148 16478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:51.443581 16478 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:46:51.444420 16497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:51.444597 16478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:46:51.444708 16478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "5e09a8ec28794781968b814593e4c8bf"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "aceda51da202173d2abbaef6bd355c9c"
server_key_iv: "dafc86ba874fe5d7853d78e98f64a5f0"
server_key_version: "encryptionkey@0"
I20260926 20:46:51.445071 16478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:51.457249 16478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:51.459123 16478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:51.459290 16478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:51.459535 16478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:51.460055 16504 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260926 20:46:51.461491 16504 ts_tablet_manager.cc:1852] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 20:46:51.461772 16504 ts_tablet_manager.cc:1905] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:46:51.464025 16504 ts_tablet_manager.cc:1918] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:46:51.464102 16504 log.cc:1196] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/2d2b9d6eb3c442a3ab1ba0ee64a03bd8
I20260926 20:46:51.464392 16504 ts_tablet_manager.cc:1939] T 2d2b9d6eb3c442a3ab1ba0ee64a03bd8 P 5e09a8ec28794781968b814593e4c8bf: Deleting consensus metadata
I20260926 20:46:51.464635 16478 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 20:46:51.464732 16478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260926 20:46:51.464783 16478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:46:51.464834 16478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:51.471246 16478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:39345
I20260926 20:46:51.471347 16611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:39345 every 8 connection(s)
I20260926 20:46:51.471645 16478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:46:51.472546 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 16478
I20260926 20:46:51.472855 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:42389
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:38437
--webserver_port=46553
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:46:51 dist-test-slave-4hd3 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:51.481884 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.472467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:42933 (local address 127.0.98.126:38437)
0926 20:46:51.472522 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:51.472525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:51.472788 (+ 263us) server_negotiation.cc:407] Connection header received
0926 20:46:51.473383 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:51.473384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:51.473400 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:51.473489 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:51.474169 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.474700 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.476252 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.476467 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.476571 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.478833 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:51.478834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:51.478835 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:51.478843 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:51.480386 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:51.480856 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:51.480857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:51.480857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:51.480922 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:51.481214 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:51.481215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:51.481216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:51.481332 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:51.481418 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:51.481698 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 20:46:51.481792 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:46:51.482103 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.472459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:42933)
0926 20:46:51.472648 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:51.472691 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:51.473273 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:51.473563 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:51.473566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:51.473818 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:51.474060 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:51.474065 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.474809 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.474810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:51.476145 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:51.476148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.476583 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.476584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:51.476721 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.477309 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:51.477318 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:51.478725 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:51.480526 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:51.480533 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:51.480536 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:51.480769 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:51.480994 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:51.480995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:51.480996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:51.481117 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:51.481402 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:51.481409 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:51.481557 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 20:46:51.481675 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 20:46:51.481832 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":116,"threads_started":1}
I20260926 20:46:51.482663 16612 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38437
I20260926 20:46:51.482765 16612 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:51.482983 16612 heartbeater.cc:511] Master 127.0.98.126:38437 requested a full tablet report, sending...
I20260926 20:46:51.483374 16285 ts_manager.cc:194] Registered new tserver with Master: 5e09a8ec28794781968b814593e4c8bf (127.0.98.66:39345)
I20260926 20:46:51.483861 16285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:42933
W20260926 20:46:51.555518 16615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:46:51.555783 16615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:46:51.555824 16615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:46:51.557003 16615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:46:51.557082 16615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:46:51.557129 16615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:46:51.558645 16615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:42389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=46553
--tserver_master_addrs=127.0.98.126:38437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:46:51.558921 16615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:46:51.559162 16615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:46:51.564518 16623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:51.564551 16621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:46:51.564518 16620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:46:51.564886 16615 server_base.cc:1034] running on GCE node
I20260926 20:46:51.565133 16615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:46:51.565342 16615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:46:51.566509 16615 hybrid_clock.cc:648] HybridClock initialized: now 1790455611566476 us; error 46 us; skew 500 ppm
Sep 26 20:46:51 dist-test-slave-4hd3 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:46:51.574792 16615 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:46:51.575297 16615 webserver.cc:460] Webserver started at http://127.0.98.67:46553/ using document root <none> and password file <none>
I20260926 20:46:51.575541 16615 fs_manager.cc:359] Metadata directory not provided
I20260926 20:46:51.575628 16615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:46:51.577090 16615 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:46:51.577761 16630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:51.578176 16615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:46:51.578320 16615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "28bc76dae6da4a66b0bbe69e485e27f2"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "77ee7967a212333bf9625e1706e4b83c"
server_key_iv: "5b3665c09bd889f3d3d52eec5a86d287"
server_key_version: "encryptionkey@0"
I20260926 20:46:51.578696 16615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:46:51.598312 16615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:46:51.600715 16615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:46:51.600835 16615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:46:51.601056 16615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:46:51.601379 16615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:46:51.601413 16615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:51.601436 16615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:46:51.601493 16615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:51.607827 16615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:42389
I20260926 20:46:51.607936 16743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:42389 every 8 connection(s)
I20260926 20:46:51.608258 16615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:46:51.608745 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 16615
Sep 26 20:46:51 dist-test-slave-4hd3 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:46:51.616323 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.609273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51462 (local address 127.0.98.126:38437)
0926 20:46:51.609377 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:51.609379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:51.609533 (+ 154us) server_negotiation.cc:407] Connection header received
0926 20:46:51.609609 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:51.609610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:51.609622 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:51.609678 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:51.610296 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.610832 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.611595 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.611901 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.612012 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.613347 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:51.613348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:51.613348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:51.613357 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:51.614966 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:51.615342 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:51.615343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:51.615343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:51.615410 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:51.615674 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:51.615675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:51.615675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:51.615858 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:51.616055 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:51.616093 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 20:46:51.616219 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":54,"threads_started":1}
I20260926 20:46:51.618733 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.609057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:59667 (local address 127.0.98.126:38437)
0926 20:46:51.609124 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:51.609126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:51.609335 (+ 209us) server_negotiation.cc:407] Connection header received
0926 20:46:51.609874 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:51.609875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:51.609894 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:51.609944 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:51.610892 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.611365 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.612725 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.612937 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.613025 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.615516 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:51.615517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:51.615517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:51.615524 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:51.617355 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:51.617758 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:51.617759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:51.617759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:51.617834 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:51.618111 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:51.618112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:51.618113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:51.618222 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:51.618299 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:51.618571 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 20:46:51.618644 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:46:51.618960 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.609052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:59667)
0926 20:46:51.609221 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:51.609262 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:51.609756 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:51.610015 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:51.610019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:51.610591 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:46:51.610778 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:51.610783 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.611479 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.611480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:51.612620 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:51.612622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.613036 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.613037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:51.613181 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.613673 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:46:51.613682 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:46:51.615427 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:46:51.617433 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:51.617439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:51.617441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:51.617661 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:51.617904 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:46:51.617904 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:46:51.617905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:46:51.618037 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:46:51.618291 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:46:51.618297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:46:51.618436 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 20:46:51.618551 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 20:46:51.618710 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260926 20:46:51.619472 16744 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38437
I20260926 20:46:51.619566 16744 heartbeater.cc:464] Registering TS with master...
I20260926 20:46:51.619753 16744 heartbeater.cc:511] Master 127.0.98.126:38437 requested a full tablet report, sending...
I20260926 20:46:51.620091 16285 ts_manager.cc:194] Registered new tserver with Master: 28bc76dae6da4a66b0bbe69e485e27f2 (127.0.98.67:42389)
I20260926 20:46:51.620584 16285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:59667
I20260926 20:46:51.628337 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:46:51.631186 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.628746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.0.98.126:38437)
0926 20:46:51.628850 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:51.628853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:51.628934 (+ 81us) server_negotiation.cc:407] Connection header received
0926 20:46:51.629003 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:51.629003 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:51.629036 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:51.629098 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:46:51.629203 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.629665 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.630458 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.630664 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.630735 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.630967 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:46:51.631009 (+ 42us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 20:46:51.631139 (+ 130us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51478: token expired
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 20:46:51.631279 16476 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51478: token expired
W20260926 20:46:51.631376 16214 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.126:38437)
I20260926 20:46:51.637506 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.631654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51482 (local address 127.0.98.126:38437)
0926 20:46:51.631756 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:51.631758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:51.631765 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:46:51.631784 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:51.631785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:51.631794 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:51.631855 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:46:51.632328 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.632754 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.633480 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.633656 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.633710 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.634855 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:46:51.634856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:46:51.634856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:46:51.634862 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:46:51.636321 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:51.636690 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:51.636691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:51.636692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:51.636747 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:46:51.636996 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:46:51.636997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:46:51.636997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:46:51.637129 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:46:51.637210 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:51.637315 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 20:46:51.637391 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:46:51.638754 16285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:46:51.640483 16285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:46:51.643056 16390 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6b7e5327ce4ebf9c30909d517114e5 (DEFAULT_TABLE table=test-table [id=6d29549169f94bbaa30920e711ffa795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:46:51.643285 16390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e6b7e5327ce4ebf9c30909d517114e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:51.645123 16365 tablet_bootstrap.cc:492] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1: Bootstrap starting.
I20260926 20:46:51.645653 16365 tablet_bootstrap.cc:654] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:51.646329 16365 tablet_bootstrap.cc:492] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1: No bootstrap required, opened a new log
I20260926 20:46:51.646413 16365 ts_tablet_manager.cc:1397] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:46:51.646579 16365 raft_consensus.cc:348] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:51.646651 16365 raft_consensus.cc:374] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:51.646687 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.642492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:42389 (local address 127.0.0.1:34484)
0926 20:46:51.642577 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:51.642590 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:51.642633 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:51.643382 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:51.643383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:51.643384 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:51.643438 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:51.643439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.644430 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.644431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:51.645403 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:51.645406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.646406 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.646407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:51.646486 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.646491 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:46:51.646562 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 20:46:51.646610 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 20:46:51.646698 16365 raft_consensus.cc:729] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9243cda34234a07abb66bcb8a40d1a1, State: Initialized, Role: FOLLOWER
I20260926 20:46:51.646709 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.642733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34484 (local address 127.0.98.67:42389)
0926 20:46:51.643113 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:51.643117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:51.643126 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:46:51.643189 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:51.643191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:51.643209 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:51.643319 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:51.643534 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.644327 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.645554 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.646311 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.646390 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.646449 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:51.646587 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 20:46:51.646628 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":81,"threads_started":1}
I20260926 20:46:51.646749 16365 consensus_queue.cc:260] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:51.646874 16365 ts_tablet_manager.cc:1428] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:46:51.647198 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.642577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39345 (local address 127.0.0.1:57352)
0926 20:46:51.642741 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:51.642751 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:51.642827 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:51.643335 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:51.643336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:51.643339 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:51.643401 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:51.643403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.644834 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.644834 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:51.645969 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:51.645971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.646953 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.646954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:51.647018 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.647023 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:46:51.647053 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 20:46:51.647133 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":117,"threads_started":1}
I20260926 20:46:51.647559 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.642951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57352 (local address 127.0.98.66:39345)
0926 20:46:51.643133 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:51.643138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:51.643151 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:46:51.643193 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:51.643195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:51.643212 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:51.643486 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:51.643581 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.644734 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.646061 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.646835 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.647306 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.647375 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:51.647409 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 20:46:51.647449 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":69,"threads_started":1}
I20260926 20:46:51.647758 16666 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6b7e5327ce4ebf9c30909d517114e5 (DEFAULT_TABLE table=test-table [id=6d29549169f94bbaa30920e711ffa795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:46:51.648157 16666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e6b7e5327ce4ebf9c30909d517114e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:51.651014 16754 tablet_bootstrap.cc:492] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2: Bootstrap starting.
I20260926 20:46:51.651585 16546 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6b7e5327ce4ebf9c30909d517114e5 (DEFAULT_TABLE table=test-table [id=6d29549169f94bbaa30920e711ffa795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:46:51.651687 16754 tablet_bootstrap.cc:654] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:51.651921 16546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e6b7e5327ce4ebf9c30909d517114e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:46:51.652269 16754 log.cc:824] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:51.653029 16754 tablet_bootstrap.cc:492] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2: No bootstrap required, opened a new log
I20260926 20:46:51.653124 16754 ts_tablet_manager.cc:1397] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:46:51.653944 16504 tablet_bootstrap.cc:492] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf: Bootstrap starting.
I20260926 20:46:51.654729 16504 tablet_bootstrap.cc:654] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf: Neither blocks nor log segments found. Creating new log.
I20260926 20:46:51.655043 16504 log.cc:824] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf: Log is configured to *not* fsync() on all Append() calls
I20260926 20:46:51.655009 16754 raft_consensus.cc:348] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:51.655184 16754 raft_consensus.cc:374] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:51.655221 16754 raft_consensus.cc:729] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28bc76dae6da4a66b0bbe69e485e27f2, State: Initialized, Role: FOLLOWER
I20260926 20:46:51.655339 16754 consensus_queue.cc:260] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:51.655658 16754 ts_tablet_manager.cc:1428] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:46:51.655678 16744 heartbeater.cc:503] Master 127.0.98.126:38437 was elected leader, sending a full tablet report...
I20260926 20:46:51.655907 16504 tablet_bootstrap.cc:492] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf: No bootstrap required, opened a new log
I20260926 20:46:51.655983 16504 ts_tablet_manager.cc:1397] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:46:51.657505 16504 raft_consensus.cc:348] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:51.657655 16504 raft_consensus.cc:374] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:46:51.657699 16504 raft_consensus.cc:729] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e09a8ec28794781968b814593e4c8bf, State: Initialized, Role: FOLLOWER
I20260926 20:46:51.657833 16504 consensus_queue.cc:260] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:51.658087 16504 ts_tablet_manager.cc:1428] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 20:46:51.658125 16612 heartbeater.cc:503] Master 127.0.98.126:38437 was elected leader, sending a full tablet report...
I20260926 20:46:51.685192 16758 raft_consensus.cc:482] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:46:51.685305 16758 raft_consensus.cc:504] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:51.685657 16758 leader_election.cc:288] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28bc76dae6da4a66b0bbe69e485e27f2 (127.0.98.67:42389), d9243cda34234a07abb66bcb8a40d1a1 (127.0.98.65:38855)
I20260926 20:46:51.688711 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.685817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:42389 (local address 127.0.98.66:38577)
0926 20:46:51.685874 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:51.685887 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:51.685933 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:51.686126 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:51.686127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:51.686128 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:51.686199 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:51.686201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.687042 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.687043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:51.687879 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:51.687882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.688448 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.688449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:51.688531 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.688534 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:46:51.688599 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 20:46:51.688637 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 20:46:51.688711 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.685863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:38577 (local address 127.0.98.67:42389)
0926 20:46:51.685972 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:51.685974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:51.685983 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:46:51.686012 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:51.686013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:51.686027 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:51.686120 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:51.686282 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.686961 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.687969 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.688353 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.688449 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.688459 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:51.688619 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 20:46:51.688655 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 20:46:51.689049 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.685970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:33325 (local address 127.0.98.65:38855)
0926 20:46:51.686021 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:51.686023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:51.686118 (+ 95us) server_negotiation.cc:407] Connection header received
0926 20:46:51.686183 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:51.686184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:51.686195 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:51.686267 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:46:51.686452 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.687113 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.688159 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.688549 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.688638 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.688647 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:51.688949 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 20:46:51.688991 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 20:46:51.689070 16698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6b7e5327ce4ebf9c30909d517114e5" candidate_uuid: "5e09a8ec28794781968b814593e4c8bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" is_pre_election: true
I20260926 20:46:51.689241 16698 raft_consensus.cc:2454] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e09a8ec28794781968b814593e4c8bf in term 0.
I20260926 20:46:51.689272 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.685828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:38855 (local address 127.0.98.66:33325)
0926 20:46:51.686045 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:51.686052 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 20:46:51.686122 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:46:51.686268 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:46:51.686269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:46:51.686271 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:46:51.686348 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:51.686349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.687199 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.687200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:51.688070 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:46:51.688073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.688639 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.688640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:46:51.688710 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.688713 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:46:51.689155 (+ 442us) client_negotiation.cc:240] Negotiation successful
0926 20:46:51.689204 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":1,"thread_start_us":151,"threads_started":1}
I20260926 20:46:51.689514 16501 leader_election.cc:302] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28bc76dae6da4a66b0bbe69e485e27f2, 5e09a8ec28794781968b814593e4c8bf; no voters:
I20260926 20:46:51.689658 16758 raft_consensus.cc:2781] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:46:51.689715 16758 raft_consensus.cc:482] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:46:51.689733 16758 raft_consensus.cc:3037] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:51.689693 16427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6b7e5327ce4ebf9c30909d517114e5" candidate_uuid: "5e09a8ec28794781968b814593e4c8bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9243cda34234a07abb66bcb8a40d1a1" is_pre_election: true
I20260926 20:46:51.689854 16427 raft_consensus.cc:2454] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e09a8ec28794781968b814593e4c8bf in term 0.
I20260926 20:46:51.690513 16758 raft_consensus.cc:504] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:51.690711 16758 leader_election.cc:288] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [CANDIDATE]: Term 1 election: Requested vote from peers 28bc76dae6da4a66b0bbe69e485e27f2 (127.0.98.67:42389), d9243cda34234a07abb66bcb8a40d1a1 (127.0.98.65:38855)
I20260926 20:46:51.690855 16427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6b7e5327ce4ebf9c30909d517114e5" candidate_uuid: "5e09a8ec28794781968b814593e4c8bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9243cda34234a07abb66bcb8a40d1a1"
I20260926 20:46:51.690872 16698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6b7e5327ce4ebf9c30909d517114e5" candidate_uuid: "5e09a8ec28794781968b814593e4c8bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28bc76dae6da4a66b0bbe69e485e27f2"
I20260926 20:46:51.690950 16698 raft_consensus.cc:3037] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:51.690955 16427 raft_consensus.cc:3037] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:46:51.691633 16698 raft_consensus.cc:2454] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e09a8ec28794781968b814593e4c8bf in term 1.
I20260926 20:46:51.691848 16501 leader_election.cc:302] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28bc76dae6da4a66b0bbe69e485e27f2, 5e09a8ec28794781968b814593e4c8bf; no voters:
I20260926 20:46:51.691991 16758 raft_consensus.cc:2781] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:46:51.692143 16758 raft_consensus.cc:686] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [term 1 LEADER]: Becoming Leader. State: Replica: 5e09a8ec28794781968b814593e4c8bf, State: Running, Role: LEADER
I20260926 20:46:51.692255 16427 raft_consensus.cc:2454] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e09a8ec28794781968b814593e4c8bf in term 1.
I20260926 20:46:51.692251 16758 consensus_queue.cc:237] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:46:51.693181 16285 catalog_manager.cc:5495] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e09a8ec28794781968b814593e4c8bf (127.0.98.66). New cstate: current_term: 1 leader_uuid: "5e09a8ec28794781968b814593e4c8bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } health_report { overall_health: UNKNOWN } } }
I20260926 20:46:51.700806 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.698406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.0.98.66:39345)
0926 20:46:51.698442 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 20:46:51.698444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:46:51.698452 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:46:51.698504 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:46:51.698505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:46:51.698524 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:46:51.698587 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:46:51.698720 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.699186 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.700005 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:46:51.700210 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:46:51.700286 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:46:51.700465 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:46:51.700564 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:46:51.700656 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:46:51.700694 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 20:46:51.700713 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 20:46:51.702497 16698 raft_consensus.cc:1261] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 1 FOLLOWER]: Refusing update from remote peer 5e09a8ec28794781968b814593e4c8bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:46:51.702526 16427 raft_consensus.cc:1261] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Refusing update from remote peer 5e09a8ec28794781968b814593e4c8bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:46:51.702834 16758 consensus_queue.cc:1035] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:46:51.702950 16760 consensus_queue.cc:1035] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:46:51.705281 16763 mvcc.cc:205] Tried to move back new op lower bound from 7333706185531019264 to 7333706185492262912. Current Snapshot: MvccSnapshot[applied={T|T < 7333706185531019264}]
I20260926 20:46:51.705875 16764 mvcc.cc:205] Tried to move back new op lower bound from 7333706185531019264 to 7333706185492262912. Current Snapshot: MvccSnapshot[applied={T|T < 7333706185531019264 or (T in {7333706185531019264})}]
I20260926 20:46:51.707934 16284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:46:51.708037 16284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:46:51.708787 16284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: Sending DeleteTablet for 3 replicas of tablet 6e6b7e5327ce4ebf9c30909d517114e5
I20260926 20:46:51.709085 16666 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e6b7e5327ce4ebf9c30909d517114e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:46:51 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:34484
I20260926 20:46:51.709154 16546 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e6b7e5327ce4ebf9c30909d517114e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:46:51 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:57352
I20260926 20:46:51.709304 16773 tablet_replica.cc:330] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2: stopping tablet replica
I20260926 20:46:51.709283 16390 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e6b7e5327ce4ebf9c30909d517114e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:46:51 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:39222
I20260926 20:46:51.709367 16482 tablet_replica.cc:330] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1: stopping tablet replica
I20260926 20:46:51.709408 16773 raft_consensus.cc:2229] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:46:51.709414 16482 raft_consensus.cc:2229] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:46:51.709465 16482 raft_consensus.cc:2258] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:46:51.709504 16773 raft_consensus.cc:2258] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:46:51.709734 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 16323
I20260926 20:46:51.709853 16482 ts_tablet_manager.cc:1905] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:46:51.709908 16773 ts_tablet_manager.cc:1905] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:46:51.710323 16774 tablet_replica.cc:330] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf: stopping tablet replica
I20260926 20:46:51.710419 16774 raft_consensus.cc:2229] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:46:51.710525 16774 raft_consensus.cc:2258] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:46:51.710947 16774 ts_tablet_manager.cc:1905] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:46:51.711571 16773 ts_tablet_manager.cc:1918] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:46:51.711633 16773 log.cc:1196] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/6e6b7e5327ce4ebf9c30909d517114e5
I20260926 20:46:51.711951 16773 ts_tablet_manager.cc:1939] T 6e6b7e5327ce4ebf9c30909d517114e5 P 28bc76dae6da4a66b0bbe69e485e27f2: Deleting consensus metadata
I20260926 20:46:51.712322 16272 catalog_manager.cc:4841] TS 28bc76dae6da4a66b0bbe69e485e27f2 (127.0.98.67:42389): tablet 6e6b7e5327ce4ebf9c30909d517114e5 (table test-table [id=6d29549169f94bbaa30920e711ffa795]) successfully deleted
I20260926 20:46:51.712621 16774 ts_tablet_manager.cc:1918] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:46:51.712658 16774 log.cc:1196] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/6e6b7e5327ce4ebf9c30909d517114e5
I20260926 20:46:51.712811 16774 ts_tablet_manager.cc:1939] T 6e6b7e5327ce4ebf9c30909d517114e5 P 5e09a8ec28794781968b814593e4c8bf: Deleting consensus metadata
I20260926 20:46:51.713608 16269 catalog_manager.cc:4841] TS 5e09a8ec28794781968b814593e4c8bf (127.0.98.66:39345): tablet 6e6b7e5327ce4ebf9c30909d517114e5 (table test-table [id=6d29549169f94bbaa30920e711ffa795]) successfully deleted
W20260926 20:46:51.717270 16269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:38855 (error 108)
I20260926 20:46:51.717782 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:46:51.717649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:38855 (local address 127.0.0.1:39226)
0926 20:46:51.717713 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 20:46:51.717754 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 20:46:51.717772 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 16478
W20260926 20:46:51.719352 16269 catalog_manager.cc:4584] TS d9243cda34234a07abb66bcb8a40d1a1 (127.0.98.65:38855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6e6b7e5327ce4ebf9c30909d517114e5: Network error: Client connection negotiation failed: client connection to 127.0.98.65:38855: connect: Connection refused (error 111)
I20260926 20:46:51.722877 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 16615
I20260926 20:46:51.727295 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 16252
I20260926 20:47:11.732904 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38437
--webserver_interface=127.0.98.126
--webserver_port=40097
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:38437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:11.814528 16776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:11.814723 16776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:11.814744 16776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:11.815883 16776 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:47:11.815922 16776 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:47:11.815941 16776 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:47:11.815953 16776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:11.815966 16776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:11.815979 16776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:11.815991 16776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:11.816004 16776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:11.817387 16776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:38437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=40097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:11.817628 16776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:11.817801 16776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:11.822788 16781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:11.822816 16784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:11.822857 16776 server_base.cc:1034] running on GCE node
W20260926 20:47:11.822816 16782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:11.823385 16776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:11.823758 16776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:11.824915 16776 hybrid_clock.cc:648] HybridClock initialized: now 1790455631824895 us; error 39 us; skew 500 ppm
Sep 26 20:47:11 dist-test-slave-4hd3 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:11.832276 16776 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:47:11.832767 16776 webserver.cc:460] Webserver started at http://127.0.98.126:40097/ using document root <none> and password file <none>
I20260926 20:47:11.832950 16776 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:11.832993 16776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:11.834306 16776 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:11.835041 16791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:11.835350 16776 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:11.835495 16776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "a60fabb67dd24bb59ec33c8673857818"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "9e3d30525cb39c28e21c8401ce659449"
server_key_iv: "f0e826300623b9c1505d6c0b00525d3b"
server_key_version: "encryptionkey@0"
I20260926 20:47:11.835903 16776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:11.854782 16776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:11.856796 16776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:11.856905 16776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:11.861094 16776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:38437
I20260926 20:47:11.861219 16843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:38437 every 8 connection(s)
I20260926 20:47:11.861544 16776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:11.862990 16844 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:47:11.863692 16844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: Bootstrap starting.
I20260926 20:47:11.868335 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 16776
I20260926 20:47:11.868667 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:38855
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:38437
--webserver_port=40231
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:47:11.872880 16844 log.cc:824] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:11.875275 16844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:47:11.875625 16844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: Bootstrap complete.
I20260926 20:47:11.877939 16844 raft_consensus.cc:348] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60fabb67dd24bb59ec33c8673857818" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38437 } }
I20260926 20:47:11.878183 16844 raft_consensus.cc:729] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60fabb67dd24bb59ec33c8673857818, State: Initialized, Role: FOLLOWER
I20260926 20:47:11.878360 16844 consensus_queue.cc:260] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60fabb67dd24bb59ec33c8673857818" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38437 } }
I20260926 20:47:11.878420 16844 raft_consensus.cc:388] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:11.878482 16844 raft_consensus.cc:482] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:11.878531 16844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:47:11.879541 16844 raft_consensus.cc:504] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60fabb67dd24bb59ec33c8673857818" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38437 } }
I20260926 20:47:11.879668 16844 leader_election.cc:302] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a60fabb67dd24bb59ec33c8673857818; no voters:
I20260926 20:47:11.879880 16844 leader_election.cc:288] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 20:47:11.879917 16848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 20:47:11.880035 16848 raft_consensus.cc:686] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [term 3 LEADER]: Becoming Leader. State: Replica: a60fabb67dd24bb59ec33c8673857818, State: Running, Role: LEADER
I20260926 20:47:11.880160 16844 sys_catalog.cc:564] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:11.880141 16848 consensus_queue.cc:237] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60fabb67dd24bb59ec33c8673857818" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38437 } }
I20260926 20:47:11.880427 16848 sys_catalog.cc:455] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a60fabb67dd24bb59ec33c8673857818. Latest consensus state: current_term: 3 leader_uuid: "a60fabb67dd24bb59ec33c8673857818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60fabb67dd24bb59ec33c8673857818" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38437 } } }
I20260926 20:47:11.880494 16848 sys_catalog.cc:458] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:11.880649 16849 sys_catalog.cc:455] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a60fabb67dd24bb59ec33c8673857818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60fabb67dd24bb59ec33c8673857818" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38437 } } }
I20260926 20:47:11.880725 16849 sys_catalog.cc:458] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:11.880846 16852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:11.881672 16852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:11.881781 16852 catalog_manager.cc:1259] Loaded cluster ID: 7a1d8ca797b44af5a4ab2a43e4c41a1e
I20260926 20:47:11.881820 16852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:47:11.882969 16852 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:11.883164 16852 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: Loaded TSK: 0
I20260926 20:47:11.883215 16852 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a60fabb67dd24bb59ec33c8673857818: Loaded TSK: 1
F20260926 20:47:11.883437 16852 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5a8262f9ad google::LogMessage::Fail() at ??:0
@ 0x7f5a8263288a google::LogMessage::SendToLog() at ??:0
@ 0x7f5a8262f4de google::LogMessage::Flush() at ??:0
@ 0x7f5a82630a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5a84d9c104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f5a84d99d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f5a8a5891cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f5a8a58948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f5a8a589594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f5a8a54fc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f5a8a559ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5a844fd0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5a844f2954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5a853566db start_thread at ??:0
@ 0x7f5a8112d61f clone at ??:0
W20260926 20:47:11.961146 16846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:11.961377 16846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:11.961414 16846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:11.962577 16846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:11.962643 16846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:11.962706 16846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:47:11.964324 16846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:38855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=40231
--tserver_master_addrs=127.0.98.126:38437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:11.964577 16846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:11.964802 16846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:11.970124 16886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:11.970124 16888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:11.970151 16885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:11.970918 16846 server_base.cc:1034] running on GCE node
I20260926 20:47:11.971150 16846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:11.971537 16846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:11.972851 16846 hybrid_clock.cc:648] HybridClock initialized: now 1790455631972836 us; error 36 us; skew 500 ppm
Sep 26 20:47:11 dist-test-slave-4hd3 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:11.975616 16846 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:47:11.976231 16846 webserver.cc:460] Webserver started at http://127.0.98.65:40231/ using document root <none> and password file <none>
I20260926 20:47:11.976470 16846 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:11.976545 16846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:11.978118 16846 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:47:11.978976 16895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:11.979207 16846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:11.979326 16846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "d9243cda34234a07abb66bcb8a40d1a1"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "25ea6f88a4842d03bbd8d8f9b9fe5023"
server_key_iv: "135b890d119c649729ff28de8cd64839"
server_key_version: "encryptionkey@0"
I20260926 20:47:11.979776 16846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:11.990532 16846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:11.992807 16846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:11.992970 16846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:11.993221 16846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:11.993655 16902 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:47:11.995687 16846 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:47:11.995781 16846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 20:47:11.995868 16846 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:47:11.996480 16846 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:47:11.996536 16846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 20:47:11.996598 16902 tablet_bootstrap.cc:492] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1: Bootstrap starting.
I20260926 20:47:11.998507 16902 log.cc:824] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:12.001906 16902 tablet_bootstrap.cc:492] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:47:12.002249 16902 tablet_bootstrap.cc:492] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1: Bootstrap complete.
I20260926 20:47:12.002413 16902 ts_tablet_manager.cc:1397] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:47:12.003666 16846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:38855
I20260926 20:47:12.003834 17010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:38855 every 8 connection(s)
I20260926 20:47:12.004067 16846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:47:12.004691 16902 raft_consensus.cc:348] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:47:12.005038 16902 raft_consensus.cc:729] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9243cda34234a07abb66bcb8a40d1a1, State: Initialized, Role: FOLLOWER
I20260926 20:47:12.005203 16902 consensus_queue.cc:260] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:47:12.005478 16902 ts_tablet_manager.cc:1428] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:47:12.014245 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 16846
I20260926 20:47:12.014521 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:39345
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:38437
--webserver_port=40745
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:12.103179 17013 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:47:12.004859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:54421)
0926 20:47:12.005140 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:12.005190 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:12.005348 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:12.102969 (+ 97621us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
W20260926 20:47:12.103469 17011 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:38437 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.126:38437: Client connection negotiation failed: client connection to 127.0.98.126:38437: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:47:12.103899 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:12.103778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:39751)
0926 20:47:12.103833 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:12.103869 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 20:47:12.104334 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:12.104248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:45469)
0926 20:47:12.104296 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:12.104319 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 20:47:12.104460 17011 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:47:12.113469 17015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:12.113698 17015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:12.113736 17015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:12.114882 17015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:12.114951 17015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:12.114998 17015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:47:12.116531 17015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:39345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=40745
--tserver_master_addrs=127.0.98.126:38437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:12.116806 17015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:12.117038 17015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:12.122262 17021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:12.122269 17023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:12.122361 17015 server_base.cc:1034] running on GCE node
W20260926 20:47:12.122273 17020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:12.122766 17015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:12.123011 17015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:12.124190 17015 hybrid_clock.cc:648] HybridClock initialized: now 1790455632124175 us; error 44 us; skew 500 ppm
Sep 26 20:47:12 dist-test-slave-4hd3 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:12.126855 17015 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:47:12.127306 17015 webserver.cc:460] Webserver started at http://127.0.98.66:40745/ using document root <none> and password file <none>
I20260926 20:47:12.127487 17015 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:12.127528 17015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:12.128854 17015 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:12.129454 17030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:12.129724 17015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:12.129812 17015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "5e09a8ec28794781968b814593e4c8bf"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "aceda51da202173d2abbaef6bd355c9c"
server_key_iv: "dafc86ba874fe5d7853d78e98f64a5f0"
server_key_version: "encryptionkey@0"
I20260926 20:47:12.130132 17015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:12.152706 17015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:12.155115 17015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:12.155233 17015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:12.155413 17015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:12.155810 17015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:12.155851 17015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:12.155874 17015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:12.155887 17015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:12.161904 17015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:39345
I20260926 20:47:12.162005 17143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:39345 every 8 connection(s)
I20260926 20:47:12.162343 17015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:47:12.163465 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:12.163089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:36133)
0926 20:47:12.163254 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:12.163326 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
W20260926 20:47:12.163882 17144 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:38437 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.126:38437: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
I20260926 20:47:12.164175 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:12.164068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:55923)
0926 20:47:12.164115 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:12.164143 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:47:12.164469 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:12.164383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:32929)
0926 20:47:12.164426 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:12.164448 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 20:47:12.164570 17144 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:47:12.170325 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 17015
I20260926 20:47:12.170620 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:42389
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:38437
--webserver_port=46553
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:12.249953 17147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:12.250339 17147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:12.250423 17147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:12.251596 17147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:12.251639 17147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:12.251667 17147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:47:12.253126 17147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:42389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=46553
--tserver_master_addrs=127.0.98.126:38437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:12.253361 17147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:12.253616 17147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:12.258607 17152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:12.258602 17153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:12.258602 17155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:12.258831 17147 server_base.cc:1034] running on GCE node
I20260926 20:47:12.259048 17147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:12.259311 17147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:12.260471 17147 hybrid_clock.cc:648] HybridClock initialized: now 1790455632260454 us; error 28 us; skew 500 ppm
Sep 26 20:47:12 dist-test-slave-4hd3 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:12.262959 17147 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:47:12.263424 17147 webserver.cc:460] Webserver started at http://127.0.98.67:46553/ using document root <none> and password file <none>
I20260926 20:47:12.263662 17147 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:12.263765 17147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:12.265092 17147 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:47:12.265758 17162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:12.265937 17147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:47:12.266040 17147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "28bc76dae6da4a66b0bbe69e485e27f2"
format_stamp: "Formatted at 2026-09-26 20:46:30 on dist-test-slave-4hd3"
server_key: "77ee7967a212333bf9625e1706e4b83c"
server_key_iv: "5b3665c09bd889f3d3d52eec5a86d287"
server_key_version: "encryptionkey@0"
I20260926 20:47:12.266394 17147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:12.279161 17147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:12.281409 17147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:12.281584 17147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:12.281836 17147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:12.282181 17147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:12.282215 17147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:12.282238 17147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:12.282250 17147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:12.288602 17147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:42389
I20260926 20:47:12.288653 17275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:42389 every 8 connection(s)
I20260926 20:47:12.288995 17147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:47:12.290035 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:12.289671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:37283)
0926 20:47:12.289825 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:12.289898 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
W20260926 20:47:12.290458 17276 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:38437 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.126:38437: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
I20260926 20:47:12.290777 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:12.290674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:47839)
0926 20:47:12.290724 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:12.290753 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:47:12.291049 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:12.290959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:42465)
0926 20:47:12.291004 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:12.291027 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 20:47:12.291153 17276 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:47:12.295287 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 17147
I20260926 20:47:13.105093 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:13.104865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:54781)
0926 20:47:13.104992 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:13.105031 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260926 20:47:13.165146 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:13.164942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:55779)
0926 20:47:13.165065 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:13.165100 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260926 20:47:13.291705 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:13.291495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:49289)
0926 20:47:13.291617 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:13.291655 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260926 20:47:13.371440 17283 raft_consensus.cc:482] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:47:13.371615 17283 raft_consensus.cc:504] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:47:13.372014 17283 leader_election.cc:288] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28bc76dae6da4a66b0bbe69e485e27f2 (127.0.98.67:42389), 5e09a8ec28794781968b814593e4c8bf (127.0.98.66:39345)
Sep 26 20:47:13 dist-test-slave-4hd3 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.67@KRBTEST.COM
Sep 26 20:47:13 dist-test-slave-4hd3 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.66@KRBTEST.COM
I20260926 20:47:13.382920 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:13.372413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:44785 (local address 127.0.98.67:42389)
0926 20:47:13.372543 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:13.372550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:13.372577 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:47:13.373090 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:13.373093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:13.373152 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:13.373306 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:13.374310 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:13.375150 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:13.376167 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:13.376565 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:13.376698 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:13.379071 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:13.379074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:13.379075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:13.379084 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:13.381099 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:13.381558 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:13.381560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:13.381560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:13.381640 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:13.381925 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:13.381926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:13.381926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:13.382249 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:13.382335 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:13.382673 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 20:47:13.382808 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":57,"threads_started":1}
I20260926 20:47:13.382920 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:13.372326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:44211 (local address 127.0.98.66:39345)
0926 20:47:13.372447 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:13.372460 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:13.372496 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:47:13.373090 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:13.373093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:13.373152 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:13.373334 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:13.374179 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:13.375035 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:13.376186 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:13.376574 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:13.376697 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:13.379468 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:13.379471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:13.379472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:13.379481 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:13.381343 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:13.381691 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:13.381694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:13.381694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:13.381811 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:13.382004 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:13.382005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:13.382006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:13.382249 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:13.382335 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:13.382673 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 20:47:13.382817 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":65,"threads_started":1}
I20260926 20:47:13.382910 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:13.372208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:42389 (local address 127.0.98.65:44785)
0926 20:47:13.372353 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:13.372364 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:13.372433 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:13.373442 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:13.373443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:13.373810 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:13.374195 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:13.374198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:13.375251 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:13.375252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:13.376076 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:13.376078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:13.376673 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:13.376674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:13.376800 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:13.377397 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:47:13.377405 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:13.378945 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:13.381194 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:13.381201 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:13.381203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:13.381477 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:13.381729 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:13.381730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:13.381730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:13.381829 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:13.382350 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:13.382351 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:13.382486 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 20:47:13.382603 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 20:47:13.382791 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":6,"thread_start_us":79,"threads_started":1}
I20260926 20:47:13.383059 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:13.372190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39345 (local address 127.0.98.65:44211)
0926 20:47:13.372247 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:13.372272 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:13.372335 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:13.373388 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:13.373391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:13.373807 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:13.374042 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:13.374047 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:13.375144 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:13.375145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:13.376077 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:13.376080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:13.376671 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:13.376672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:13.376801 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:13.377450 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:47:13.377454 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:13.379355 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:13.381420 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:13.381421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:13.381421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:13.381620 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:13.381876 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:13.381877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:13.381877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:13.381936 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:13.382328 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:13.382335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:13.382486 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 20:47:13.382602 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 20:47:13.382865 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":202}
W20260926 20:47:13.383860 16896 leader_election.cc:341] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e09a8ec28794781968b814593e4c8bf (127.0.98.66:39345): Not found: Tablet not found: 6e6b7e5327ce4ebf9c30909d517114e5
W20260926 20:47:13.384456 16899 leader_election.cc:341] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28bc76dae6da4a66b0bbe69e485e27f2 (127.0.98.67:42389): Not found: Tablet not found: 6e6b7e5327ce4ebf9c30909d517114e5
I20260926 20:47:13.384502 16899 leader_election.cc:302] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9243cda34234a07abb66bcb8a40d1a1; no voters: 28bc76dae6da4a66b0bbe69e485e27f2, 5e09a8ec28794781968b814593e4c8bf
I20260926 20:47:13.384604 17283 raft_consensus.cc:2726] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:47:14.105842 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:14.105573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:54735)
0926 20:47:14.105753 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:14.105795 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":83,"threads_started":1}
I20260926 20:47:14.165944 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:14.165703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:47469)
0926 20:47:14.165857 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:14.165900 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":72,"threads_started":1}
I20260926 20:47:14.292625 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:14.292356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:33445)
0926 20:47:14.292521 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:14.292579 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
I20260926 20:47:15.106588 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:15.106358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:53905)
0926 20:47:15.106504 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:15.106542 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260926 20:47:15.166855 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:15.166612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:43941)
0926 20:47:15.166760 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:15.166803 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
I20260926 20:47:15.178324 17292 raft_consensus.cc:482] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:47:15.178418 17292 raft_consensus.cc:504] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:47:15.178637 17292 leader_election.cc:288] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28bc76dae6da4a66b0bbe69e485e27f2 (127.0.98.67:42389), 5e09a8ec28794781968b814593e4c8bf (127.0.98.66:39345)
W20260926 20:47:15.179056 16896 leader_election.cc:341] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e09a8ec28794781968b814593e4c8bf (127.0.98.66:39345): Not found: Tablet not found: 6e6b7e5327ce4ebf9c30909d517114e5
W20260926 20:47:15.179139 16899 leader_election.cc:341] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28bc76dae6da4a66b0bbe69e485e27f2 (127.0.98.67:42389): Not found: Tablet not found: 6e6b7e5327ce4ebf9c30909d517114e5
I20260926 20:47:15.179225 16899 leader_election.cc:302] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9243cda34234a07abb66bcb8a40d1a1; no voters: 28bc76dae6da4a66b0bbe69e485e27f2, 5e09a8ec28794781968b814593e4c8bf
I20260926 20:47:15.179314 17292 raft_consensus.cc:2726] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:47:15.293439 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:15.293204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:39357)
0926 20:47:15.293355 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:15.293397 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260926 20:47:16.107512 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:16.107250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:50849)
0926 20:47:16.107413 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:16.107462 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
I20260926 20:47:16.167666 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:16.167409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:43375)
0926 20:47:16.167555 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:16.167617 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":61,"threads_started":1}
I20260926 20:47:16.294234 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:16.293973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:37641)
0926 20:47:16.294113 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:16.294191 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260926 20:47:17.108279 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:17.108029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:38931)
0926 20:47:17.108192 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:17.108232 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":63,"threads_started":1}
I20260926 20:47:17.168414 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:17.168209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:41063)
0926 20:47:17.168334 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:17.168369 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260926 20:47:17.294920 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:17.294716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:38803)
0926 20:47:17.294843 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:17.294878 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260926 20:47:17.886554 17300 raft_consensus.cc:482] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:47:17.886650 17300 raft_consensus.cc:504] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:47:17.886824 17300 leader_election.cc:288] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28bc76dae6da4a66b0bbe69e485e27f2 (127.0.98.67:42389), 5e09a8ec28794781968b814593e4c8bf (127.0.98.66:39345)
W20260926 20:47:17.887204 16899 leader_election.cc:341] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28bc76dae6da4a66b0bbe69e485e27f2 (127.0.98.67:42389): Not found: Tablet not found: 6e6b7e5327ce4ebf9c30909d517114e5
W20260926 20:47:17.887282 16896 leader_election.cc:341] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e09a8ec28794781968b814593e4c8bf (127.0.98.66:39345): Not found: Tablet not found: 6e6b7e5327ce4ebf9c30909d517114e5
I20260926 20:47:17.887343 16899 leader_election.cc:302] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9243cda34234a07abb66bcb8a40d1a1; no voters: 28bc76dae6da4a66b0bbe69e485e27f2, 5e09a8ec28794781968b814593e4c8bf
I20260926 20:47:17.887435 17300 raft_consensus.cc:2726] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:47:18.108991 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:18.108771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:47245)
0926 20:47:18.108901 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:18.108941 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260926 20:47:18.169265 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:18.169025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:35117)
0926 20:47:18.169173 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:18.169211 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260926 20:47:18.295822 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:18.295462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:46183)
0926 20:47:18.295632 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:18.295702 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260926 20:47:19.109658 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:19.109440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:50141)
0926 20:47:19.109573 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:19.109610 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260926 20:47:19.169939 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:19.169724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:55419)
0926 20:47:19.169851 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:19.169895 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260926 20:47:19.296598 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:19.296352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:59033)
0926 20:47:19.296511 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:19.296554 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":64,"threads_started":1}
I20260926 20:47:20.110324 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:20.110119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:53943)
0926 20:47:20.110244 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:20.110279 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260926 20:47:20.170815 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:20.170425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:54729)
0926 20:47:20.170561 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:20.170624 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
I20260926 20:47:20.297395 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:20.297131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:42347)
0926 20:47:20.297301 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:20.297352 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
I20260926 20:47:21.111052 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:21.110840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:39969)
0926 20:47:21.110967 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:21.111006 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260926 20:47:21.171546 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:21.171317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:48683)
0926 20:47:21.171457 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:21.171501 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260926 20:47:21.298094 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:21.297885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:58569)
0926 20:47:21.298012 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:21.298051 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260926 20:47:22.111897 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:22.111624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:39783)
0926 20:47:22.111813 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:22.111850 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
I20260926 20:47:22.172273 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:22.172065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:39461)
0926 20:47:22.172186 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:22.172229 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260926 20:47:22.298882 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:22.298656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:52319)
0926 20:47:22.298790 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:22.298836 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260926 20:47:22.356011 17316 raft_consensus.cc:482] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:47:22.356108 17316 raft_consensus.cc:504] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc76dae6da4a66b0bbe69e485e27f2" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42389 } } peers { permanent_uuid: "5e09a8ec28794781968b814593e4c8bf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39345 } } peers { permanent_uuid: "d9243cda34234a07abb66bcb8a40d1a1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38855 } }
I20260926 20:47:22.356316 17316 leader_election.cc:288] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28bc76dae6da4a66b0bbe69e485e27f2 (127.0.98.67:42389), 5e09a8ec28794781968b814593e4c8bf (127.0.98.66:39345)
W20260926 20:47:22.356668 16899 leader_election.cc:341] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28bc76dae6da4a66b0bbe69e485e27f2 (127.0.98.67:42389): Not found: Tablet not found: 6e6b7e5327ce4ebf9c30909d517114e5
W20260926 20:47:22.356766 16896 leader_election.cc:341] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e09a8ec28794781968b814593e4c8bf (127.0.98.66:39345): Not found: Tablet not found: 6e6b7e5327ce4ebf9c30909d517114e5
I20260926 20:47:22.356868 16896 leader_election.cc:302] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9243cda34234a07abb66bcb8a40d1a1; no voters: 28bc76dae6da4a66b0bbe69e485e27f2, 5e09a8ec28794781968b814593e4c8bf
I20260926 20:47:22.356982 17316 raft_consensus.cc:2726] T 6e6b7e5327ce4ebf9c30909d517114e5 P d9243cda34234a07abb66bcb8a40d1a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:47:23.112726 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:23.112463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:44613)
0926 20:47:23.112622 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:23.112667 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":69,"threads_started":1}
I20260926 20:47:23.172981 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:23.172771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:39957)
0926 20:47:23.172894 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:23.172935 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260926 20:47:23.299659 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:23.299437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:52461)
0926 20:47:23.299554 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:23.299599 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260926 20:47:24.113413 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:24.113201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:38509)
0926 20:47:24.113329 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:24.113367 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260926 20:47:24.173679 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:24.173439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:33707)
0926 20:47:24.173591 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:24.173633 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260926 20:47:24.300385 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:24.300160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:33261)
0926 20:47:24.300299 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:24.300341 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260926 20:47:25.114102 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:25.113871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:53009)
0926 20:47:25.114018 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:25.114056 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
I20260926 20:47:25.174401 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:25.174175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:45729)
0926 20:47:25.174322 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:25.174357 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
I20260926 20:47:25.301082 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:25.300874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:41261)
0926 20:47:25.300994 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:25.301036 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260926 20:47:26.114756 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:26.114527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:49259)
0926 20:47:26.114677 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:26.114710 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260926 20:47:26.175161 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:26.174898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:45137)
0926 20:47:26.175040 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:26.175096 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260926 20:47:26.301796 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:26.301580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:51947)
0926 20:47:26.301719 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:26.301752 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
I20260926 20:47:27.115466 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:27.115242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.65:49991)
0926 20:47:27.115374 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:27.115419 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260926 20:47:27.175954 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:27.175710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.66:55155)
0926 20:47:27.175877 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:27.175911 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
I20260926 20:47:27.298388 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 16846
I20260926 20:47:27.302538 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:27.302301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38437 (local address 127.0.98.67:43553)
0926 20:47:27.302434 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:27.302480 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":53,"threads_started":1}
I20260926 20:47:27.303345 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 17015
I20260926 20:47:27.307638 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 17147
2026-09-26T20:47:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58293 ms)
[----------] 30 tests from SecurityITest (230691 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:27 dist-test-slave-4hd3 krb5kdc[17334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:27 dist-test-slave-4hd3 krb5kdc[17334](info): set up 2 sockets
Sep 26 20:47:27 dist-test-slave-4hd3 krb5kdc[17334](info): commencing operation
krb5kdc: starting...
W20260926 20:47:28.343456 393 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:28 dist-test-slave-4hd3 krb5kdc[17334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:47:28.459666 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34687
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:34687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:28.539996 17347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:28.540202 17347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:28.540225 17347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:28.541385 17347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:28.541430 17347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:28.541445 17347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:28.541457 17347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:28.541491 17347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:28.542968 17347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:34687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:28.543231 17347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:28.543476 17347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:47:28.548303 17347 server_base.cc:1034] running on GCE node
W20260926 20:47:28.548265 17352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:28.548383 17353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:28.548265 17355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:28.548739 17347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:28.549038 17347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:28.550212 17347 hybrid_clock.cc:648] HybridClock initialized: now 1790455648550203 us; error 37 us; skew 500 ppm
Sep 26 20:47:28 dist-test-slave-4hd3 krb5kdc[17334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:28.552690 17347 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:47:28.553068 17347 webserver.cc:460] Webserver started at http://127.0.98.126:44313/ using document root <none> and password file <none>
I20260926 20:47:28.553236 17347 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:28.553275 17347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:28.553362 17347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:28.554396 17347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "090f6f0b7e20440188a977c8e6deda8a"
format_stamp: "Formatted at 2026-09-26 20:47:28 on dist-test-slave-4hd3"
server_key: "c1fa02ac32a2351132fb077bcf10aefd"
server_key_iv: "e2fd3786a8df7a12182d7087b05a4b34"
server_key_version: "encryptionkey@0"
I20260926 20:47:28.554692 17347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "090f6f0b7e20440188a977c8e6deda8a"
format_stamp: "Formatted at 2026-09-26 20:47:28 on dist-test-slave-4hd3"
server_key: "c1fa02ac32a2351132fb077bcf10aefd"
server_key_iv: "e2fd3786a8df7a12182d7087b05a4b34"
server_key_version: "encryptionkey@0"
I20260926 20:47:28.555969 17347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:47:28.556668 17362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:28.556856 17347 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:47:28.556954 17347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "090f6f0b7e20440188a977c8e6deda8a"
format_stamp: "Formatted at 2026-09-26 20:47:28 on dist-test-slave-4hd3"
server_key: "c1fa02ac32a2351132fb077bcf10aefd"
server_key_iv: "e2fd3786a8df7a12182d7087b05a4b34"
server_key_version: "encryptionkey@0"
I20260926 20:47:28.557024 17347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:28.581600 17347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:28.583513 17347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:28.583673 17347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:28.587805 17347 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:34687
I20260926 20:47:28.587860 17414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:34687 every 8 connection(s)
I20260926 20:47:28.588230 17347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:28.589133 17415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:28.591396 17415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a: Bootstrap starting.
I20260926 20:47:28.592022 17415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:28.592257 17415 log.cc:824] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:28.592962 17415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a: No bootstrap required, opened a new log
I20260926 20:47:28.594281 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 17347
I20260926 20:47:28.594401 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:28.594592 393 external_mini_cluster.cc:1428] Setting key ebd0288618881f3b18d12d51e53a84d7
I20260926 20:47:28.594621 17415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "090f6f0b7e20440188a977c8e6deda8a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34687 } }
I20260926 20:47:28.594820 17415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:28.594870 17415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 090f6f0b7e20440188a977c8e6deda8a, State: Initialized, Role: FOLLOWER
I20260926 20:47:28.595307 17415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "090f6f0b7e20440188a977c8e6deda8a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34687 } }
I20260926 20:47:28.595412 17415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:28.595458 17415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:28.595512 17415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:28.596696 17415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "090f6f0b7e20440188a977c8e6deda8a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34687 } }
I20260926 20:47:28.596813 17415 leader_election.cc:302] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 090f6f0b7e20440188a977c8e6deda8a; no voters:
I20260926 20:47:28.596985 17415 leader_election.cc:288] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:28.597024 17420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:28.597172 17420 raft_consensus.cc:686] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a [term 1 LEADER]: Becoming Leader. State: Replica: 090f6f0b7e20440188a977c8e6deda8a, State: Running, Role: LEADER
I20260926 20:47:28.597244 17415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:28.597357 17420 consensus_queue.cc:237] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "090f6f0b7e20440188a977c8e6deda8a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34687 } }
I20260926 20:47:28.597729 17420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 090f6f0b7e20440188a977c8e6deda8a. Latest consensus state: current_term: 1 leader_uuid: "090f6f0b7e20440188a977c8e6deda8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "090f6f0b7e20440188a977c8e6deda8a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34687 } } }
I20260926 20:47:28.597743 17422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "090f6f0b7e20440188a977c8e6deda8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "090f6f0b7e20440188a977c8e6deda8a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34687 } } }
I20260926 20:47:28.597913 17422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:28.598115 17420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:28.598304 17429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:28.598816 17429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:28.600654 17429 catalog_manager.cc:1347] Generated new cluster ID: 60f011e7bb7249aeaf9b50ec27ad4592
I20260926 20:47:28.600705 17429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:47:28 dist-test-slave-4hd3 krb5kdc[17334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:47:28.605916 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:28.595093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55016 (local address 127.0.98.126:34687)
0926 20:47:28.595291 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:28.595302 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:28.595333 (+ 31us) server_negotiation.cc:407] Connection header received
0926 20:47:28.595443 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:28.595450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:28.595510 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:28.595664 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:28.596475 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:28.597741 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:28.598919 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:28.599510 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:28.599623 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:28.602574 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:28.602578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:28.602579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:28.602591 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:28.604429 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:28.604791 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:28.604794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:28.604794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:28.604897 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:28.605097 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:28.605098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:28.605099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:28.605340 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:28.605502 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:28.605602 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 20:47:28.605746 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":109,"threads_started":1}
I20260926 20:47:28.618844 17429 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:28.619446 17429 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:28.628530 17429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a: Generated new TSK 0
I20260926 20:47:28.628732 17429 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260926 20:47:28.658689 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34687
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:28.742615 17441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:28.742825 17441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:28.742849 17441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:28.744140 17441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:28.744184 17441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:28.744228 17441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:47:28.745756 17441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:28.746011 17441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:28.746241 17441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:28.751140 17446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:28.751140 17449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:28.751209 17447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:28.751520 17441 server_base.cc:1034] running on GCE node
I20260926 20:47:28.751698 17441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:28.752022 17441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:28.753172 17441 hybrid_clock.cc:648] HybridClock initialized: now 1790455648753160 us; error 26 us; skew 500 ppm
Sep 26 20:47:28 dist-test-slave-4hd3 krb5kdc[17334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:28.755702 17441 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:47:28.756201 17441 webserver.cc:460] Webserver started at http://127.0.98.65:46769/ using document root <none> and password file <none>
I20260926 20:47:28.756402 17441 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:28.756444 17441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:28.756533 17441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:28.757515 17441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "fe0a8bb1003c42d88c8f8b47bf84de2b"
format_stamp: "Formatted at 2026-09-26 20:47:28 on dist-test-slave-4hd3"
server_key: "3befb99afdc0223f3d321dd2b83ac62b"
server_key_iv: "493d71485f6790899fdffc913fe67ec9"
server_key_version: "encryptionkey@0"
I20260926 20:47:28.757808 17441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "fe0a8bb1003c42d88c8f8b47bf84de2b"
format_stamp: "Formatted at 2026-09-26 20:47:28 on dist-test-slave-4hd3"
server_key: "3befb99afdc0223f3d321dd2b83ac62b"
server_key_iv: "493d71485f6790899fdffc913fe67ec9"
server_key_version: "encryptionkey@0"
I20260926 20:47:28.759057 17441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:47:28.759856 17456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:28.760010 17441 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:47:28.760098 17441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "fe0a8bb1003c42d88c8f8b47bf84de2b"
format_stamp: "Formatted at 2026-09-26 20:47:28 on dist-test-slave-4hd3"
server_key: "3befb99afdc0223f3d321dd2b83ac62b"
server_key_iv: "493d71485f6790899fdffc913fe67ec9"
server_key_version: "encryptionkey@0"
I20260926 20:47:28.760152 17441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:28.770735 17441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:28.772646 17441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:28.772755 17441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:28.772965 17441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:28.773259 17441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:28.773291 17441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:28.773340 17441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:28.773365 17441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:28.778928 17441 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:41545
I20260926 20:47:28.778981 17569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:41545 every 8 connection(s)
I20260926 20:47:28.779306 17441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:47:28.783990 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 17441
I20260926 20:47:28.784092 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:47:28.784317 393 external_mini_cluster.cc:1428] Setting key 11c593b0d7ea0815171837f89210ec01
Sep 26 20:47:28 dist-test-slave-4hd3 krb5kdc[17334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:47:28.789490 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:28.780149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:59537 (local address 127.0.98.126:34687)
0926 20:47:28.780223 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:28.780226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:28.780449 (+ 223us) server_negotiation.cc:407] Connection header received
0926 20:47:28.781362 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:28.781363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:28.781377 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:28.781435 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:28.782039 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:28.782535 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:28.783479 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:28.783672 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:28.783826 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:28.786460 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:28.786461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:28.786461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:28.786467 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:28.788047 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:28.788469 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:28.788470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:28.788470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:28.788535 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:28.788800 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:28.788801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:28.788801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:28.788921 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:28.788990 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:28.789312 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 20:47:28.789401 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:47:28.789683 17572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:28.780111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34687 (local address 127.0.98.65:59537)
0926 20:47:28.780296 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:28.780332 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:28.781264 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:28.781511 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:28.781515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:28.781780 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:28.781964 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:28.781968 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:28.782611 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:28.782612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:28.783405 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:28.783407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:28.783823 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:28.783824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:28.783919 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:28.784461 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:47:28.784485 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:28.786364 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:28.788118 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:28.788130 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:28.788132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:28.788361 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:28.788601 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:28.788602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:28.788602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:28.788705 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:28.788991 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:28.788998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:28.789171 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 20:47:28.789289 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 20:47:28.789443 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":111,"threads_started":1}
I20260926 20:47:28.790177 17570 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34687
I20260926 20:47:28.790289 17570 heartbeater.cc:464] Registering TS with master...
I20260926 20:47:28.790472 17570 heartbeater.cc:511] Master 127.0.98.126:34687 requested a full tablet report, sending...
I20260926 20:47:28.791107 17379 ts_manager.cc:194] Registered new tserver with Master: fe0a8bb1003c42d88c8f8b47bf84de2b (127.0.98.65:41545)
I20260926 20:47:28.791944 17379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:59537
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260926 20:47:28.808573 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34687
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:28.887868 17575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:28.888057 17575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:28.888080 17575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:28.889217 17575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:28.889258 17575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:28.889307 17575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:47:28.890792 17575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:28.891038 17575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:28.891263 17575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:28.896008 17583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:28.896025 17581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:28.896052 17580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:28.896163 17575 server_base.cc:1034] running on GCE node
I20260926 20:47:28.896469 17575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:28.896700 17575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:28.897908 17575 hybrid_clock.cc:648] HybridClock initialized: now 1790455648897889 us; error 27 us; skew 500 ppm
Sep 26 20:47:28 dist-test-slave-4hd3 krb5kdc[17334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:28.900382 17575 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:47:28.900811 17575 webserver.cc:460] Webserver started at http://127.0.98.66:35541/ using document root <none> and password file <none>
I20260926 20:47:28.901000 17575 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:28.901039 17575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:28.901125 17575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:28.902494 17575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "3e38f94b6ca44241b17b0597d02e9347"
format_stamp: "Formatted at 2026-09-26 20:47:28 on dist-test-slave-4hd3"
server_key: "477918e50bd0cdc22aea0e5bbcb69fed"
server_key_iv: "010c3d9268eba5b7704a7533263880d0"
server_key_version: "encryptionkey@0"
I20260926 20:47:28.902937 17575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "3e38f94b6ca44241b17b0597d02e9347"
format_stamp: "Formatted at 2026-09-26 20:47:28 on dist-test-slave-4hd3"
server_key: "477918e50bd0cdc22aea0e5bbcb69fed"
server_key_iv: "010c3d9268eba5b7704a7533263880d0"
server_key_version: "encryptionkey@0"
I20260926 20:47:28.904752 17575 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260926 20:47:28.905725 17590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:28.905926 17575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:47:28.906018 17575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "3e38f94b6ca44241b17b0597d02e9347"
format_stamp: "Formatted at 2026-09-26 20:47:28 on dist-test-slave-4hd3"
server_key: "477918e50bd0cdc22aea0e5bbcb69fed"
server_key_iv: "010c3d9268eba5b7704a7533263880d0"
server_key_version: "encryptionkey@0"
I20260926 20:47:28.906111 17575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:28.917564 17575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:28.919446 17575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:28.919593 17575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:28.919850 17575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:28.920132 17575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:28.920181 17575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:28.920221 17575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:28.920248 17575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:28.925792 17575 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:43651
I20260926 20:47:28.925828 17703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:43651 every 8 connection(s)
I20260926 20:47:28.926146 17575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:47:28 dist-test-slave-4hd3 krb5kdc[17334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:47:28.932912 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 17575
I20260926 20:47:28.933009 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:47:28.933229 393 external_mini_cluster.cc:1428] Setting key 6d5332cf21fae7e800c02471969cb5c7
I20260926 20:47:28.935599 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:28.926829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:41899 (local address 127.0.98.126:34687)
0926 20:47:28.926873 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:28.926875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:28.927087 (+ 212us) server_negotiation.cc:407] Connection header received
0926 20:47:28.928024 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:28.928025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:28.928037 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:28.928100 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:28.928754 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:28.929255 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:28.930234 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:28.930442 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:28.930498 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:28.932604 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:28.932606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:28.932607 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:28.932615 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:28.934109 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:28.934593 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:28.934594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:28.934594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:28.934644 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:28.934901 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:28.934902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:28.934902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:28.934999 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:28.935047 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:28.935389 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 20:47:28.935478 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:47:28.935685 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:28.926801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34687 (local address 127.0.98.66:41899)
0926 20:47:28.926966 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:28.927006 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:28.927925 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:28.928172 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:28.928175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:28.928486 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:28.928672 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:28.928677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:28.929330 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:28.929330 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:28.930162 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:28.930164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:28.930513 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:28.930514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:28.930614 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:28.931111 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:47:28.931118 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:28.932505 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:28.934186 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:28.934192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:28.934195 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:28.934511 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:28.934701 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:28.934702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:28.934702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:28.934840 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:28.935056 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:28.935061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:28.935228 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 20:47:28.935357 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 20:47:28.935502 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260926 20:47:28.936187 17704 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34687
I20260926 20:47:28.936302 17704 heartbeater.cc:464] Registering TS with master...
I20260926 20:47:28.936474 17704 heartbeater.cc:511] Master 127.0.98.126:34687 requested a full tablet report, sending...
I20260926 20:47:28.936833 17379 ts_manager.cc:194] Registered new tserver with Master: 3e38f94b6ca44241b17b0597d02e9347 (127.0.98.66:43651)
I20260926 20:47:28.937209 17379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:41899
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260926 20:47:28.958626 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34687
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:29.038846 17709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:29.039041 17709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:29.039063 17709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:29.040135 17709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:29.040167 17709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:29.040192 17709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:47:29.041569 17709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:29.041749 17709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:29.041987 17709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:47:29.047979 17709 server_base.cc:1034] running on GCE node
W20260926 20:47:29.047937 17714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:29.047931 17715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:29.047936 17717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:29.048347 17709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:29.048614 17709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:29.049774 17709 hybrid_clock.cc:648] HybridClock initialized: now 1790455649049757 us; error 31 us; skew 500 ppm
Sep 26 20:47:29 dist-test-slave-4hd3 krb5kdc[17334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:29.052330 17709 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:47:29.052744 17709 webserver.cc:460] Webserver started at http://127.0.98.67:36559/ using document root <none> and password file <none>
I20260926 20:47:29.052968 17709 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:29.053013 17709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:29.053149 17709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:29.054183 17709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "af02461d2ee443469b1f57f8bdad27e4"
format_stamp: "Formatted at 2026-09-26 20:47:29 on dist-test-slave-4hd3"
server_key: "931268bd7887d0d4e9dc3be7cc58b8cc"
server_key_iv: "bfaa6c78eb27d64fc4b362460e44a619"
server_key_version: "encryptionkey@0"
I20260926 20:47:29.054530 17709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "af02461d2ee443469b1f57f8bdad27e4"
format_stamp: "Formatted at 2026-09-26 20:47:29 on dist-test-slave-4hd3"
server_key: "931268bd7887d0d4e9dc3be7cc58b8cc"
server_key_iv: "bfaa6c78eb27d64fc4b362460e44a619"
server_key_version: "encryptionkey@0"
I20260926 20:47:29.055892 17709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:47:29.056595 17724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:29.056754 17709 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:47:29.056836 17709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "af02461d2ee443469b1f57f8bdad27e4"
format_stamp: "Formatted at 2026-09-26 20:47:29 on dist-test-slave-4hd3"
server_key: "931268bd7887d0d4e9dc3be7cc58b8cc"
server_key_iv: "bfaa6c78eb27d64fc4b362460e44a619"
server_key_version: "encryptionkey@0"
I20260926 20:47:29.056914 17709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:29.073920 17709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:29.075865 17709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:29.076017 17709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:29.076226 17709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:29.076524 17709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:29.076556 17709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:29.076596 17709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:29.076620 17709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:29.082175 17709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:42903
I20260926 20:47:29.082244 17837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:42903 every 8 connection(s)
I20260926 20:47:29.082549 17709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:47:29.083913 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 17709
I20260926 20:47:29.084079 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:47:29.084295 393 external_mini_cluster.cc:1428] Setting key b938429752adfafec3f611cde67292e6
Sep 26 20:47:29 dist-test-slave-4hd3 krb5kdc[17334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:47:29.092885 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.083356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:46261 (local address 127.0.98.126:34687)
0926 20:47:29.083427 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:29.083430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:29.083642 (+ 212us) server_negotiation.cc:407] Connection header received
0926 20:47:29.084901 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:29.084902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:29.084914 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:29.084970 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:29.085665 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.086139 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.087116 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.087336 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.087400 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.089551 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:29.089552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:29.089552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:29.089559 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:29.091266 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:29.091891 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:29.091892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:29.091892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:29.091946 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:29.092199 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:29.092199 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:29.092200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:29.092298 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:29.092381 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:29.092667 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 20:47:29.092774 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:47:29.093001 17840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.083344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34687 (local address 127.0.98.67:46261)
0926 20:47:29.083526 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:29.083562 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:29.084799 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:29.085052 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:29.085056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:29.085392 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:29.085587 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.085592 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.086214 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.086214 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.087020 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.087022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.087430 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.087431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.087528 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.088029 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:47:29.088037 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:29.089455 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:29.091351 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:29.091357 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:29.091358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:29.091804 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:29.092031 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:29.092032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:29.092032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:29.092135 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:29.092379 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:29.092384 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:29.092501 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 20:47:29.092624 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 20:47:29.092778 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
I20260926 20:47:29.093428 17838 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34687
I20260926 20:47:29.093503 17838 heartbeater.cc:464] Registering TS with master...
I20260926 20:47:29.093678 17838 heartbeater.cc:511] Master 127.0.98.126:34687 requested a full tablet report, sending...
I20260926 20:47:29.094013 17379 ts_manager.cc:194] Registered new tserver with Master: af02461d2ee443469b1f57f8bdad27e4 (127.0.98.67:42903)
I20260926 20:47:29.094391 17379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:46261
I20260926 20:47:29.096166 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:47:29.103621 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.097621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55024 (local address 127.0.98.126:34687)
0926 20:47:29.097677 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:29.097679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:29.097815 (+ 136us) server_negotiation.cc:407] Connection header received
0926 20:47:29.097898 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:29.097898 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:29.097906 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:29.097960 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:29.098495 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.098994 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.099696 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.099936 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.099992 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.101156 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:29.101157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:29.101158 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:29.101165 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:29.102499 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:29.102852 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:29.102853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:29.102854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:29.102905 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:29.103105 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:29.103106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:29.103106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:29.103243 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:29.103346 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:29.103447 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 20:47:29.103523 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:47:29.105144 17379 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:47:29.106612 17379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:47:29.112947 17849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.108862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:42903 (local address 127.0.0.1:48812)
0926 20:47:29.109158 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:29.109168 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:29.109221 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:29.109536 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:29.109537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:29.109538 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:29.109577 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.109579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.110861 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.110862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.112031 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.112033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.112652 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.112653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.112729 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.112741 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 20:47:29.112825 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:47:29.112878 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":40,"thread_start_us":41,"threads_started":1}
I20260926 20:47:29.112957 17851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.109014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48812 (local address 127.0.98.67:42903)
0926 20:47:29.109153 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:29.109157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:29.109209 (+ 52us) server_negotiation.cc:407] Connection header received
0926 20:47:29.109283 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:29.109285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:29.109299 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:29.109365 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:29.109676 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.110763 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.112093 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.112558 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.112640 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.112699 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:29.112840 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 20:47:29.112879 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260926 20:47:29.113263 17850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.108834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41714 (local address 127.0.98.66:43651)
0926 20:47:29.108995 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:29.108999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:29.109014 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:47:29.109057 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:29.109059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:29.109075 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:29.109153 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:29.109445 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.110466 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.112046 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.112767 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.112852 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.112910 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:29.113140 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 20:47:29.113189 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260926 20:47:29.113101 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.108684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41545 (local address 127.0.0.1:47598)
0926 20:47:29.108995 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:29.109006 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:29.109060 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:29.109429 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:29.109430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:29.109432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:29.109479 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.109481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.110882 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.110883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.111976 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.111978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.112871 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.112872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.112949 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.112953 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:47:29.112983 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 20:47:29.113024 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":62,"threads_started":1}
I20260926 20:47:29.113430 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.108798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.0.98.65:41545)
0926 20:47:29.109193 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:29.109197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:29.109205 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:47:29.109280 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:29.109283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:29.109300 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:29.109430 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:29.109584 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.110751 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.112076 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.112767 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.113187 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.113249 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:29.113274 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 20:47:29.113335 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":79,"threads_started":1}
I20260926 20:47:29.113194 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.108684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:43651 (local address 127.0.0.1:41714)
0926 20:47:29.108858 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:29.108899 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:29.108966 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:29.109251 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:29.109254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:29.109256 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:29.109358 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.109360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.110566 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.110567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.111961 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.111964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.112941 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.112942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.112998 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.113000 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:47:29.113043 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 20:47:29.113075 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260926 20:47:29.114063 17772 tablet_service.cc:1467] Processing CreateTablet for tablet 9598e08b9ac64bea9cc1826cc6855b6d (DEFAULT_TABLE table=test-table [id=27691621d5a34355badee7c216ab8509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:47:29.114126 17504 tablet_service.cc:1467] Processing CreateTablet for tablet 9598e08b9ac64bea9cc1826cc6855b6d (DEFAULT_TABLE table=test-table [id=27691621d5a34355badee7c216ab8509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:47:29.114156 17638 tablet_service.cc:1467] Processing CreateTablet for tablet 9598e08b9ac64bea9cc1826cc6855b6d (DEFAULT_TABLE table=test-table [id=27691621d5a34355badee7c216ab8509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:47:29.114403 17772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9598e08b9ac64bea9cc1826cc6855b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:29.114431 17638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9598e08b9ac64bea9cc1826cc6855b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:29.114423 17504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9598e08b9ac64bea9cc1826cc6855b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:29.116905 17852 tablet_bootstrap.cc:492] T 9598e08b9ac64bea9cc1826cc6855b6d P fe0a8bb1003c42d88c8f8b47bf84de2b: Bootstrap starting.
I20260926 20:47:29.117565 17852 tablet_bootstrap.cc:654] T 9598e08b9ac64bea9cc1826cc6855b6d P fe0a8bb1003c42d88c8f8b47bf84de2b: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:29.117619 17853 tablet_bootstrap.cc:492] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347: Bootstrap starting.
I20260926 20:47:29.118221 17852 log.cc:824] T 9598e08b9ac64bea9cc1826cc6855b6d P fe0a8bb1003c42d88c8f8b47bf84de2b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:29.118355 17853 tablet_bootstrap.cc:654] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:29.118866 17853 log.cc:824] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:29.119028 17852 tablet_bootstrap.cc:492] T 9598e08b9ac64bea9cc1826cc6855b6d P fe0a8bb1003c42d88c8f8b47bf84de2b: No bootstrap required, opened a new log
I20260926 20:47:29.119088 17852 ts_tablet_manager.cc:1397] T 9598e08b9ac64bea9cc1826cc6855b6d P fe0a8bb1003c42d88c8f8b47bf84de2b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:47:29.119194 17854 tablet_bootstrap.cc:492] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4: Bootstrap starting.
I20260926 20:47:29.119856 17854 tablet_bootstrap.cc:654] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:29.120420 17854 log.cc:824] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:29.121084 17854 tablet_bootstrap.cc:492] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4: No bootstrap required, opened a new log
I20260926 20:47:29.121163 17854 ts_tablet_manager.cc:1397] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 20:47:29.121227 17852 raft_consensus.cc:348] T 9598e08b9ac64bea9cc1826cc6855b6d P fe0a8bb1003c42d88c8f8b47bf84de2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e38f94b6ca44241b17b0597d02e9347" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43651 } } peers { permanent_uuid: "af02461d2ee443469b1f57f8bdad27e4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42903 } } peers { permanent_uuid: "fe0a8bb1003c42d88c8f8b47bf84de2b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41545 } }
I20260926 20:47:29.121388 17852 raft_consensus.cc:374] T 9598e08b9ac64bea9cc1826cc6855b6d P fe0a8bb1003c42d88c8f8b47bf84de2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:29.121412 17852 raft_consensus.cc:729] T 9598e08b9ac64bea9cc1826cc6855b6d P fe0a8bb1003c42d88c8f8b47bf84de2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0a8bb1003c42d88c8f8b47bf84de2b, State: Initialized, Role: FOLLOWER
I20260926 20:47:29.121546 17852 consensus_queue.cc:260] T 9598e08b9ac64bea9cc1826cc6855b6d P fe0a8bb1003c42d88c8f8b47bf84de2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e38f94b6ca44241b17b0597d02e9347" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43651 } } peers { permanent_uuid: "af02461d2ee443469b1f57f8bdad27e4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42903 } } peers { permanent_uuid: "fe0a8bb1003c42d88c8f8b47bf84de2b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41545 } }
I20260926 20:47:29.121814 17852 ts_tablet_manager.cc:1428] T 9598e08b9ac64bea9cc1826cc6855b6d P fe0a8bb1003c42d88c8f8b47bf84de2b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:47:29.121836 17570 heartbeater.cc:503] Master 127.0.98.126:34687 was elected leader, sending a full tablet report...
I20260926 20:47:29.122418 17853 tablet_bootstrap.cc:492] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347: No bootstrap required, opened a new log
I20260926 20:47:29.122498 17853 ts_tablet_manager.cc:1397] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 20:47:29.123052 17854 raft_consensus.cc:348] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e38f94b6ca44241b17b0597d02e9347" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43651 } } peers { permanent_uuid: "af02461d2ee443469b1f57f8bdad27e4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42903 } } peers { permanent_uuid: "fe0a8bb1003c42d88c8f8b47bf84de2b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41545 } }
I20260926 20:47:29.123231 17854 raft_consensus.cc:374] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:29.123281 17854 raft_consensus.cc:729] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af02461d2ee443469b1f57f8bdad27e4, State: Initialized, Role: FOLLOWER
I20260926 20:47:29.123385 17854 consensus_queue.cc:260] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e38f94b6ca44241b17b0597d02e9347" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43651 } } peers { permanent_uuid: "af02461d2ee443469b1f57f8bdad27e4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42903 } } peers { permanent_uuid: "fe0a8bb1003c42d88c8f8b47bf84de2b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41545 } }
I20260926 20:47:29.123652 17854 ts_tablet_manager.cc:1428] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 20:47:29.123684 17838 heartbeater.cc:503] Master 127.0.98.126:34687 was elected leader, sending a full tablet report...
I20260926 20:47:29.124297 17853 raft_consensus.cc:348] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e38f94b6ca44241b17b0597d02e9347" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43651 } } peers { permanent_uuid: "af02461d2ee443469b1f57f8bdad27e4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42903 } } peers { permanent_uuid: "fe0a8bb1003c42d88c8f8b47bf84de2b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41545 } }
I20260926 20:47:29.124442 17853 raft_consensus.cc:374] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:29.124486 17853 raft_consensus.cc:729] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e38f94b6ca44241b17b0597d02e9347, State: Initialized, Role: FOLLOWER
I20260926 20:47:29.124590 17853 consensus_queue.cc:260] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e38f94b6ca44241b17b0597d02e9347" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43651 } } peers { permanent_uuid: "af02461d2ee443469b1f57f8bdad27e4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42903 } } peers { permanent_uuid: "fe0a8bb1003c42d88c8f8b47bf84de2b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41545 } }
I20260926 20:47:29.124843 17853 ts_tablet_manager.cc:1428] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:47:29.124871 17704 heartbeater.cc:503] Master 127.0.98.126:34687 was elected leader, sending a full tablet report...
W20260926 20:47:29.176680 17705 tablet.cc:2369] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:47:29.211025 17860 raft_consensus.cc:482] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:47:29.211145 17860 raft_consensus.cc:504] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e38f94b6ca44241b17b0597d02e9347" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43651 } } peers { permanent_uuid: "af02461d2ee443469b1f57f8bdad27e4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42903 } } peers { permanent_uuid: "fe0a8bb1003c42d88c8f8b47bf84de2b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41545 } }
I20260926 20:47:29.211639 17860 leader_election.cc:288] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af02461d2ee443469b1f57f8bdad27e4 (127.0.98.67:42903), fe0a8bb1003c42d88c8f8b47bf84de2b (127.0.98.65:41545)
I20260926 20:47:29.214534 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.211960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:41229 (local address 127.0.98.65:41545)
0926 20:47:29.212043 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:29.212045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:29.212123 (+ 78us) server_negotiation.cc:407] Connection header received
0926 20:47:29.212157 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:29.212158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:29.212169 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:29.212228 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:29.212389 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.212829 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.213756 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.214156 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.214241 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.214250 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:29.214412 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 20:47:29.214454 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:47:29.214545 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.211844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:42903 (local address 127.0.98.66:37951)
0926 20:47:29.211907 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:29.211920 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:29.211963 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:29.212135 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:29.212136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:29.212139 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:29.212222 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.212225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.212890 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.212891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.213664 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.213666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.214226 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.214227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.214350 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.214354 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:47:29.214425 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 20:47:29.214462 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 20:47:29.214560 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.211844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41545 (local address 127.0.98.66:41229)
0926 20:47:29.212043 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:29.212054 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:29.212114 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:29.212239 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:29.212240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:29.212241 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:29.212322 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.212323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.212927 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.212928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.213666 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.213667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.214257 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.214258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.214338 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.214342 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:47:29.214385 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 20:47:29.214439 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":24,"thread_start_us":111,"threads_started":1}
I20260926 20:47:29.214551 17851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.211925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:37951 (local address 127.0.98.67:42903)
0926 20:47:29.211974 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:29.211977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:29.211986 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:47:29.212019 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:29.212020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:29.212030 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:29.212121 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:29.212302 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.212809 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.213776 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.214147 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.214199 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.214224 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:29.214444 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 20:47:29.214486 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:47:29.214864 17524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9598e08b9ac64bea9cc1826cc6855b6d" candidate_uuid: "3e38f94b6ca44241b17b0597d02e9347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0a8bb1003c42d88c8f8b47bf84de2b" is_pre_election: true
I20260926 20:47:29.214903 17792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9598e08b9ac64bea9cc1826cc6855b6d" candidate_uuid: "3e38f94b6ca44241b17b0597d02e9347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af02461d2ee443469b1f57f8bdad27e4" is_pre_election: true
I20260926 20:47:29.215013 17524 raft_consensus.cc:2454] T 9598e08b9ac64bea9cc1826cc6855b6d P fe0a8bb1003c42d88c8f8b47bf84de2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e38f94b6ca44241b17b0597d02e9347 in term 0.
I20260926 20:47:29.215075 17792 raft_consensus.cc:2454] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e38f94b6ca44241b17b0597d02e9347 in term 0.
I20260926 20:47:29.215221 17593 leader_election.cc:302] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e38f94b6ca44241b17b0597d02e9347, fe0a8bb1003c42d88c8f8b47bf84de2b; no voters:
I20260926 20:47:29.215397 17860 raft_consensus.cc:2781] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:47:29.215471 17860 raft_consensus.cc:482] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:47:29.215502 17860 raft_consensus.cc:3037] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:29.216205 17860 raft_consensus.cc:504] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e38f94b6ca44241b17b0597d02e9347" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43651 } } peers { permanent_uuid: "af02461d2ee443469b1f57f8bdad27e4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42903 } } peers { permanent_uuid: "fe0a8bb1003c42d88c8f8b47bf84de2b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41545 } }
I20260926 20:47:29.216358 17860 leader_election.cc:288] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [CANDIDATE]: Term 1 election: Requested vote from peers af02461d2ee443469b1f57f8bdad27e4 (127.0.98.67:42903), fe0a8bb1003c42d88c8f8b47bf84de2b (127.0.98.65:41545)
I20260926 20:47:29.216536 17524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9598e08b9ac64bea9cc1826cc6855b6d" candidate_uuid: "3e38f94b6ca44241b17b0597d02e9347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0a8bb1003c42d88c8f8b47bf84de2b"
I20260926 20:47:29.216536 17792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9598e08b9ac64bea9cc1826cc6855b6d" candidate_uuid: "3e38f94b6ca44241b17b0597d02e9347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af02461d2ee443469b1f57f8bdad27e4"
I20260926 20:47:29.216615 17524 raft_consensus.cc:3037] T 9598e08b9ac64bea9cc1826cc6855b6d P fe0a8bb1003c42d88c8f8b47bf84de2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:29.216617 17792 raft_consensus.cc:3037] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:29.217519 17792 raft_consensus.cc:2454] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e38f94b6ca44241b17b0597d02e9347 in term 1.
I20260926 20:47:29.217519 17524 raft_consensus.cc:2454] T 9598e08b9ac64bea9cc1826cc6855b6d P fe0a8bb1003c42d88c8f8b47bf84de2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e38f94b6ca44241b17b0597d02e9347 in term 1.
I20260926 20:47:29.217698 17593 leader_election.cc:302] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e38f94b6ca44241b17b0597d02e9347, fe0a8bb1003c42d88c8f8b47bf84de2b; no voters:
I20260926 20:47:29.217789 17860 raft_consensus.cc:2781] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:29.217931 17860 raft_consensus.cc:686] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [term 1 LEADER]: Becoming Leader. State: Replica: 3e38f94b6ca44241b17b0597d02e9347, State: Running, Role: LEADER
I20260926 20:47:29.218017 17860 consensus_queue.cc:237] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e38f94b6ca44241b17b0597d02e9347" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43651 } } peers { permanent_uuid: "af02461d2ee443469b1f57f8bdad27e4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42903 } } peers { permanent_uuid: "fe0a8bb1003c42d88c8f8b47bf84de2b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41545 } }
I20260926 20:47:29.218863 17379 catalog_manager.cc:5495] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e38f94b6ca44241b17b0597d02e9347 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "3e38f94b6ca44241b17b0597d02e9347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e38f94b6ca44241b17b0597d02e9347" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af02461d2ee443469b1f57f8bdad27e4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe0a8bb1003c42d88c8f8b47bf84de2b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41545 } health_report { overall_health: UNKNOWN } } }
I20260926 20:47:29.230244 17850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.227750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41726 (local address 127.0.98.66:43651)
0926 20:47:29.227878 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:29.227880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:29.227940 (+ 60us) server_negotiation.cc:407] Connection header received
0926 20:47:29.228040 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:29.228041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:29.228060 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:29.228129 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:47:29.228262 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.228729 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.229480 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.229683 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.229754 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.229927 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:47:29.230026 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:47:29.230115 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:29.230135 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 20:47:29.230154 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 20:47:29.232103 17524 raft_consensus.cc:1261] T 9598e08b9ac64bea9cc1826cc6855b6d P fe0a8bb1003c42d88c8f8b47bf84de2b [term 1 FOLLOWER]: Refusing update from remote peer 3e38f94b6ca44241b17b0597d02e9347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:47:29.232103 17792 raft_consensus.cc:1261] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4 [term 1 FOLLOWER]: Refusing update from remote peer 3e38f94b6ca44241b17b0597d02e9347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:47:29.232501 17862 consensus_queue.cc:1035] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af02461d2ee443469b1f57f8bdad27e4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 42903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:47:29.232626 17860 consensus_queue.cc:1035] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe0a8bb1003c42d88c8f8b47bf84de2b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:47:29.234910 17865 mvcc.cc:205] Tried to move back new op lower bound from 7333706339251732480 to 7333706339197448192. Current Snapshot: MvccSnapshot[applied={T|T < 7333706339251732480}]
I20260926 20:47:29.235096 17867 mvcc.cc:205] Tried to move back new op lower bound from 7333706339251732480 to 7333706339197448192. Current Snapshot: MvccSnapshot[applied={T|T < 7333706339251732480}]
I20260926 20:47:29.235363 17866 mvcc.cc:205] Tried to move back new op lower bound from 7333706339251732480 to 7333706339197448192. Current Snapshot: MvccSnapshot[applied={T|T < 7333706339251732480}]
I20260926 20:47:29.237076 17379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:47:29.237154 17379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:47:29.237812 17379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 090f6f0b7e20440188a977c8e6deda8a: Sending DeleteTablet for 3 replicas of tablet 9598e08b9ac64bea9cc1826cc6855b6d
I20260926 20:47:29.238142 17638 tablet_service.cc:1514] Processing DeleteTablet for tablet 9598e08b9ac64bea9cc1826cc6855b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:47:29 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:41714
I20260926 20:47:29.238153 17772 tablet_service.cc:1514] Processing DeleteTablet for tablet 9598e08b9ac64bea9cc1826cc6855b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:47:29 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:48812
I20260926 20:47:29.238145 17504 tablet_service.cc:1514] Processing DeleteTablet for tablet 9598e08b9ac64bea9cc1826cc6855b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:47:29 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:47598
I20260926 20:47:29.238471 17877 tablet_replica.cc:330] T 9598e08b9ac64bea9cc1826cc6855b6d P fe0a8bb1003c42d88c8f8b47bf84de2b: stopping tablet replica
I20260926 20:47:29.238582 17876 tablet_replica.cc:330] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347: stopping tablet replica
I20260926 20:47:29.238590 17877 raft_consensus.cc:2229] T 9598e08b9ac64bea9cc1826cc6855b6d P fe0a8bb1003c42d88c8f8b47bf84de2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:47:29.238669 17876 raft_consensus.cc:2229] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:47:29.238726 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 17441
I20260926 20:47:29.238760 17876 raft_consensus.cc:2258] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:47:29.238845 17877 raft_consensus.cc:2258] T 9598e08b9ac64bea9cc1826cc6855b6d P fe0a8bb1003c42d88c8f8b47bf84de2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:47:29.238915 17875 tablet_replica.cc:330] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4: stopping tablet replica
I20260926 20:47:29.239030 17875 raft_consensus.cc:2229] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:47:29.239138 17875 raft_consensus.cc:2258] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:47:29.241080 17875 ts_tablet_manager.cc:1905] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:47:29.242619 17875 ts_tablet_manager.cc:1918] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:47:29.242697 17875 log.cc:1196] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/9598e08b9ac64bea9cc1826cc6855b6d
I20260926 20:47:29.242960 17875 ts_tablet_manager.cc:1939] T 9598e08b9ac64bea9cc1826cc6855b6d P af02461d2ee443469b1f57f8bdad27e4: Deleting consensus metadata
I20260926 20:47:29.243351 17366 catalog_manager.cc:4841] TS af02461d2ee443469b1f57f8bdad27e4 (127.0.98.67:42903): tablet 9598e08b9ac64bea9cc1826cc6855b6d (table test-table [id=27691621d5a34355badee7c216ab8509]) successfully deleted
W20260926 20:47:29.246218 17365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:41545 (error 108)
I20260926 20:47:29.246656 17876 ts_tablet_manager.cc:1905] T 9598e08b9ac64bea9cc1826cc6855b6d P 3e38f94b6ca44241b17b0597d02e9347: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:47:29.246819 17849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.246683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41545 (local address 127.0.0.1:47608)
0926 20:47:29.246744 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:29.246786 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:41545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 20:47:29.246937 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 17575
W20260926 20:47:29.248714 17365 catalog_manager.cc:4584] TS fe0a8bb1003c42d88c8f8b47bf84de2b (127.0.98.65:41545): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9598e08b9ac64bea9cc1826cc6855b6d: Network error: Client connection negotiation failed: client connection to 127.0.98.65:41545: connect: Connection refused (error 111)
I20260926 20:47:29.251335 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 17709
I20260926 20:47:29.251420 17849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.251307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:43651 (local address 127.0.0.1:41736)
0926 20:47:29.251357 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:29.251383 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.66:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 20:47:29.256024 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 17347
2026-09-26T20:47:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1947 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T20:47:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:29Z Disabled control of system clock
I20260926 20:47:29.283687 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34399
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:45243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:34399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:47:29.362998 17882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:29.363204 17882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:29.363224 17882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:29.364303 17882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:29.364341 17882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:29.364353 17882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:29.364365 17882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:29.364377 17882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:29.365641 17882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:45243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:34399
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:29.365824 17882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:29.366052 17882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:29.370755 17888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:29.370785 17882 server_base.cc:1034] running on GCE node
W20260926 20:47:29.370810 17887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:29.370755 17890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:29.371246 17882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:29.371541 17882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:29.372706 17882 hybrid_clock.cc:648] HybridClock initialized: now 1790455649372681 us; error 37 us; skew 500 ppm
I20260926 20:47:29.374040 17882 webserver.cc:460] Webserver started at http://127.0.98.126:46057/ using document root <none> and password file <none>
I20260926 20:47:29.374289 17882 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:29.374337 17882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:29.374496 17882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:29.375535 17882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "1a0aa532caf045fdb9e8d7c516ddaad5"
format_stamp: "Formatted at 2026-09-26 20:47:29 on dist-test-slave-4hd3"
server_key: "be86f54f1a7fbb511632fefc03125ebf"
server_key_iv: "7252133f22c40f91c09a948aac4f0a0e"
server_key_version: "encryptionkey@0"
I20260926 20:47:29.375917 17882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "1a0aa532caf045fdb9e8d7c516ddaad5"
format_stamp: "Formatted at 2026-09-26 20:47:29 on dist-test-slave-4hd3"
server_key: "be86f54f1a7fbb511632fefc03125ebf"
server_key_iv: "7252133f22c40f91c09a948aac4f0a0e"
server_key_version: "encryptionkey@0"
I20260926 20:47:29.377243 17882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:47:29.378021 17896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:29.378249 17882 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:47:29.378364 17882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "1a0aa532caf045fdb9e8d7c516ddaad5"
format_stamp: "Formatted at 2026-09-26 20:47:29 on dist-test-slave-4hd3"
server_key: "be86f54f1a7fbb511632fefc03125ebf"
server_key_iv: "7252133f22c40f91c09a948aac4f0a0e"
server_key_version: "encryptionkey@0"
I20260926 20:47:29.378438 17882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:29.394007 17882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:29.394295 17882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:29.394438 17882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:29.398319 17882 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:34399
I20260926 20:47:29.398351 17948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:34399 every 8 connection(s)
I20260926 20:47:29.398669 17882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:29.399642 17949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:29.401849 17949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5: Bootstrap starting.
I20260926 20:47:29.402362 17949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:29.402602 17949 log.cc:824] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:29.403432 17949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5: No bootstrap required, opened a new log
I20260926 20:47:29.405042 17949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0aa532caf045fdb9e8d7c516ddaad5" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34399 } }
I20260926 20:47:29.405177 17949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:29.405200 17949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a0aa532caf045fdb9e8d7c516ddaad5, State: Initialized, Role: FOLLOWER
I20260926 20:47:29.405306 17949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0aa532caf045fdb9e8d7c516ddaad5" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34399 } }
I20260926 20:47:29.405385 17949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:29.405419 17949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:29.405473 17949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:29.405966 17949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0aa532caf045fdb9e8d7c516ddaad5" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34399 } }
I20260926 20:47:29.406081 17949 leader_election.cc:302] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a0aa532caf045fdb9e8d7c516ddaad5; no voters:
I20260926 20:47:29.406253 17949 leader_election.cc:288] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:29.406363 17952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:29.406517 17949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:29.406524 17952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5 [term 1 LEADER]: Becoming Leader. State: Replica: 1a0aa532caf045fdb9e8d7c516ddaad5, State: Running, Role: LEADER
I20260926 20:47:29.406752 17952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0aa532caf045fdb9e8d7c516ddaad5" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34399 } }
I20260926 20:47:29.407246 17953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a0aa532caf045fdb9e8d7c516ddaad5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0aa532caf045fdb9e8d7c516ddaad5" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34399 } } }
I20260926 20:47:29.407338 17953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:29.407342 17954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a0aa532caf045fdb9e8d7c516ddaad5. Latest consensus state: current_term: 1 leader_uuid: "1a0aa532caf045fdb9e8d7c516ddaad5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0aa532caf045fdb9e8d7c516ddaad5" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34399 } } }
I20260926 20:47:29.407505 17954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:29.407791 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 17882
I20260926 20:47:29.407855 17961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:29.407886 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:29.408090 393 external_mini_cluster.cc:1428] Setting key 94acdf653055917b3c18d4d629387495
I20260926 20:47:29.408352 17961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:29.409904 17961 catalog_manager.cc:1347] Generated new cluster ID: d11c4e2340704d3daf6900d8abd48bc3
I20260926 20:47:29.409986 17961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:47:29.414220 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.408544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.0.98.126:34399)
0926 20:47:29.408945 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:29.408953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:29.408986 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:47:29.409098 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:29.409104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:29.409153 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:29.409296 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:29.409626 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.410968 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.412079 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.412712 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.412854 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.413519 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:29.413522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:29.413523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:29.413733 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:29.413784 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:29.413799 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:29.413868 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:29.413931 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 20:47:29.414009 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":71,"threads_started":1}
I20260926 20:47:29.428154 17961 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:29.428838 17961 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:29.434800 17961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5: Generated new TSK 0
I20260926 20:47:29.434973 17961 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:29.438642 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34399
--builtin_ntp_servers=127.0.98.84:45243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:47:29.518736 17973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:29.518965 17973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:29.519001 17973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:29.520210 17973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:29.520262 17973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:29.520303 17973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:47:29.521637 17973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:45243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:29.521888 17973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:29.522169 17973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:47:29.528227 17973 server_base.cc:1034] running on GCE node
W20260926 20:47:29.528198 17981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:29.528194 17978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:29.528311 17979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:29.528626 17973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:29.528883 17973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:29.530027 17973 hybrid_clock.cc:648] HybridClock initialized: now 1790455649530014 us; error 31 us; skew 500 ppm
I20260926 20:47:29.531324 17973 webserver.cc:460] Webserver started at http://127.0.98.65:45761/ using document root <none> and password file <none>
I20260926 20:47:29.531579 17973 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:29.531625 17973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:29.531793 17973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:29.532866 17973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "e79fb0d48dda4ed2a581298cba48aaa3"
format_stamp: "Formatted at 2026-09-26 20:47:29 on dist-test-slave-4hd3"
server_key: "ff55c53cdace78a65e0e1d0200480cb2"
server_key_iv: "34d86fec1341d460f8befca8d4352960"
server_key_version: "encryptionkey@0"
I20260926 20:47:29.533224 17973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "e79fb0d48dda4ed2a581298cba48aaa3"
format_stamp: "Formatted at 2026-09-26 20:47:29 on dist-test-slave-4hd3"
server_key: "ff55c53cdace78a65e0e1d0200480cb2"
server_key_iv: "34d86fec1341d460f8befca8d4352960"
server_key_version: "encryptionkey@0"
I20260926 20:47:29.534512 17973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:47:29.535257 17987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:29.535416 17973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:47:29.535524 17973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "e79fb0d48dda4ed2a581298cba48aaa3"
format_stamp: "Formatted at 2026-09-26 20:47:29 on dist-test-slave-4hd3"
server_key: "ff55c53cdace78a65e0e1d0200480cb2"
server_key_iv: "34d86fec1341d460f8befca8d4352960"
server_key_version: "encryptionkey@0"
I20260926 20:47:29.535602 17973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:29.558241 17973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:29.558590 17973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:29.558743 17973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:29.558969 17973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:29.559305 17973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:29.559355 17973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:29.559394 17973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:29.559422 17973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:29.565080 17973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:37327
I20260926 20:47:29.565128 18100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:37327 every 8 connection(s)
I20260926 20:47:29.565445 17973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:47:29.570575 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.566153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:34333 (local address 127.0.98.126:34399)
0926 20:47:29.566220 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:29.566223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:29.566435 (+ 212us) server_negotiation.cc:407] Connection header received
0926 20:47:29.566903 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:29.566904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:29.566915 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:29.566987 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:29.567777 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.568280 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.569269 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.569456 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.569506 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.570339 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:29.570340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:29.570340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:29.570345 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:29.570358 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:29.570362 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:29.570405 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:29.570490 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 20:47:29.570516 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:47:29.570806 18103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.566116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34399 (local address 127.0.98.65:34333)
0926 20:47:29.566306 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:29.566347 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:29.566814 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:29.567054 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:29.567058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:29.567066 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:29.567647 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.567653 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.568396 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.568397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.569191 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.569193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.569527 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.569528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.569640 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.570209 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:47:29.570209 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:29.570238 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:47:29.570262 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:29.570422 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:29.570426 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:29.570428 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:47:29.570480 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 20:47:29.570560 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260926 20:47:29.571396 18101 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34399
I20260926 20:47:29.571496 18101 heartbeater.cc:464] Registering TS with master...
I20260926 20:47:29.571669 18101 heartbeater.cc:511] Master 127.0.98.126:34399 requested a full tablet report, sending...
I20260926 20:47:29.572196 17913 ts_manager.cc:194] Registered new tserver with Master: e79fb0d48dda4ed2a581298cba48aaa3 (127.0.98.65:37327)
I20260926 20:47:29.572798 17913 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.65:34333
I20260926 20:47:29.573544 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 17973
I20260926 20:47:29.573639 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:47:29.573843 393 external_mini_cluster.cc:1428] Setting key d57fef16f0e4528c742437282a622698
I20260926 20:47:29.574740 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34399
--builtin_ntp_servers=127.0.98.84:45243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:47:29.655151 18104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:29.655352 18104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:29.655373 18104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:29.656538 18104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:29.656576 18104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:29.656600 18104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:47:29.657932 18104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:45243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:29.658175 18104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:29.658525 18104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:29.663342 18112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:29.663424 18110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:29.663560 18104 server_base.cc:1034] running on GCE node
W20260926 20:47:29.663343 18109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:29.663878 18104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:29.664077 18104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:29.665251 18104 hybrid_clock.cc:648] HybridClock initialized: now 1790455649665235 us; error 29 us; skew 500 ppm
I20260926 20:47:29.666635 18104 webserver.cc:460] Webserver started at http://127.0.98.66:39869/ using document root <none> and password file <none>
I20260926 20:47:29.666924 18104 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:29.666983 18104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:29.667158 18104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:29.668372 18104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "6f8b2efe9cbe4739872d90c0c5151845"
format_stamp: "Formatted at 2026-09-26 20:47:29 on dist-test-slave-4hd3"
server_key: "500b28fce5065b0741cbb30068e81326"
server_key_iv: "bdc7ac162c817ffeb8381e2b0390d79f"
server_key_version: "encryptionkey@0"
I20260926 20:47:29.668751 18104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "6f8b2efe9cbe4739872d90c0c5151845"
format_stamp: "Formatted at 2026-09-26 20:47:29 on dist-test-slave-4hd3"
server_key: "500b28fce5065b0741cbb30068e81326"
server_key_iv: "bdc7ac162c817ffeb8381e2b0390d79f"
server_key_version: "encryptionkey@0"
I20260926 20:47:29.670112 18104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:47:29.670909 18118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:29.671094 18104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:29.671187 18104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "6f8b2efe9cbe4739872d90c0c5151845"
format_stamp: "Formatted at 2026-09-26 20:47:29 on dist-test-slave-4hd3"
server_key: "500b28fce5065b0741cbb30068e81326"
server_key_iv: "bdc7ac162c817ffeb8381e2b0390d79f"
server_key_version: "encryptionkey@0"
I20260926 20:47:29.671267 18104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:29.694715 18104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:29.695060 18104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:29.695217 18104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:29.695463 18104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:29.695892 18104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:29.695950 18104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:29.695997 18104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:29.696028 18104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:29.702353 18104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:45495
I20260926 20:47:29.702399 18231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:45495 every 8 connection(s)
I20260926 20:47:29.702742 18104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:47:29.708065 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.703472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:40305 (local address 127.0.98.126:34399)
0926 20:47:29.703525 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:29.703528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:29.703823 (+ 295us) server_negotiation.cc:407] Connection header received
0926 20:47:29.704316 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:29.704318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:29.704334 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:29.704405 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:29.705045 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.705583 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.706609 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.706847 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.706897 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.707836 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:29.707837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:29.707838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:29.707842 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:29.707855 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:29.707859 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:29.707907 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:29.707975 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 20:47:29.708001 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:47:29.708245 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.703474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34399 (local address 127.0.98.66:40305)
0926 20:47:29.703675 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:29.703710 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:29.704230 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:29.704467 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:29.704470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:29.704478 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:29.704966 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.704970 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.705667 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.705668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.706541 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.706543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.706924 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.706925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.707038 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.707651 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:47:29.707652 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:29.707680 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:47:29.707702 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:29.707909 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:29.707913 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:29.707915 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:47:29.707966 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 20:47:29.708042 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
I20260926 20:47:29.708715 18232 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34399
I20260926 20:47:29.708814 18232 heartbeater.cc:464] Registering TS with master...
I20260926 20:47:29.708989 18232 heartbeater.cc:511] Master 127.0.98.126:34399 requested a full tablet report, sending...
I20260926 20:47:29.709378 17913 ts_manager.cc:194] Registered new tserver with Master: 6f8b2efe9cbe4739872d90c0c5151845 (127.0.98.66:45495)
I20260926 20:47:29.709573 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 18104
I20260926 20:47:29.709681 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:47:29.709908 393 external_mini_cluster.cc:1428] Setting key 7a2102d6cf2c712d6be1992a42c2390c
I20260926 20:47:29.709959 17913 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.66:40305
I20260926 20:47:29.710868 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34399
--builtin_ntp_servers=127.0.98.84:45243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:47:29.790531 18235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:29.790750 18235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:29.790783 18235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:29.791908 18235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:29.791965 18235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:29.792009 18235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:47:29.793341 18235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:45243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:29.793594 18235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:29.793874 18235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:29.798930 18243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:29.798969 18241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:29.798929 18240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:29.798962 18235 server_base.cc:1034] running on GCE node
I20260926 20:47:29.799402 18235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:29.799592 18235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:29.800724 18235 hybrid_clock.cc:648] HybridClock initialized: now 1790455649800712 us; error 27 us; skew 500 ppm
I20260926 20:47:29.801995 18235 webserver.cc:460] Webserver started at http://127.0.98.67:38197/ using document root <none> and password file <none>
I20260926 20:47:29.802235 18235 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:29.802282 18235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:29.802402 18235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:29.803442 18235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "0d6a9185440248859d01f2c9796f90dd"
format_stamp: "Formatted at 2026-09-26 20:47:29 on dist-test-slave-4hd3"
server_key: "a4357dab92c3b8d469baf83d5441d49e"
server_key_iv: "759b0807e8738c3db82df16738bc404a"
server_key_version: "encryptionkey@0"
I20260926 20:47:29.803818 18235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "0d6a9185440248859d01f2c9796f90dd"
format_stamp: "Formatted at 2026-09-26 20:47:29 on dist-test-slave-4hd3"
server_key: "a4357dab92c3b8d469baf83d5441d49e"
server_key_iv: "759b0807e8738c3db82df16738bc404a"
server_key_version: "encryptionkey@0"
I20260926 20:47:29.805095 18235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:29.805750 18249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:29.805979 18235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:47:29.806082 18235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "0d6a9185440248859d01f2c9796f90dd"
format_stamp: "Formatted at 2026-09-26 20:47:29 on dist-test-slave-4hd3"
server_key: "a4357dab92c3b8d469baf83d5441d49e"
server_key_iv: "759b0807e8738c3db82df16738bc404a"
server_key_version: "encryptionkey@0"
I20260926 20:47:29.806154 18235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:29.817062 18235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:29.817361 18235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:29.817512 18235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:29.817724 18235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:29.818044 18235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:29.818094 18235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:29.818128 18235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:29.818154 18235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:29.823637 18235 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:35705
I20260926 20:47:29.823707 18362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:35705 every 8 connection(s)
I20260926 20:47:29.824065 18235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:47:29.825305 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 18235
I20260926 20:47:29.825408 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:47:29.825609 393 external_mini_cluster.cc:1428] Setting key 8e1f5781b8e992fe4390d2177e6bfeb4
I20260926 20:47:29.829768 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.824840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:41383 (local address 127.0.98.126:34399)
0926 20:47:29.824940 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:29.824943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:29.825153 (+ 210us) server_negotiation.cc:407] Connection header received
0926 20:47:29.825913 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:29.825914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:29.825941 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:29.826006 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:29.826825 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.827340 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.828469 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.828654 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.828720 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.829530 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:29.829531 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:29.829531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:29.829535 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:29.829548 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:29.829552 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:29.829605 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:29.829687 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 20:47:29.829709 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:47:29.829952 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.824796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34399 (local address 127.0.98.67:41383)
0926 20:47:29.825008 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:29.825050 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:29.825768 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:29.826078 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:29.826082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:29.826093 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:29.826743 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.826748 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.827437 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.827438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.828372 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.828374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.828743 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.828744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.828865 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.829378 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:47:29.829379 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:29.829407 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:47:29.829429 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:29.829611 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:29.829615 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:29.829618 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:47:29.829671 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 20:47:29.829745 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260926 20:47:29.830415 18363 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34399
I20260926 20:47:29.830492 18363 heartbeater.cc:464] Registering TS with master...
I20260926 20:47:29.830629 18363 heartbeater.cc:511] Master 127.0.98.126:34399 requested a full tablet report, sending...
I20260926 20:47:29.830935 17913 ts_manager.cc:194] Registered new tserver with Master: 0d6a9185440248859d01f2c9796f90dd (127.0.98.67:35705)
I20260926 20:47:29.831296 17913 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.67:41383
I20260926 20:47:29.837148 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:47:29.841555 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.838543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.0.98.126:34399)
0926 20:47:29.838610 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:29.838612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:29.838737 (+ 125us) server_negotiation.cc:407] Connection header received
0926 20:47:29.838840 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:29.838841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:29.838851 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:29.838901 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:29.839062 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.839545 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.840341 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.840541 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.840619 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.841243 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:29.841245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:29.841246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:29.841251 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:29.841265 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:29.841269 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:29.841382 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:29.841445 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 20:47:29.841469 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:47:29.842994 17913 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:47:29.844663 17913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:47:29.851052 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.847174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:37327 (local address 127.0.0.1:36630)
0926 20:47:29.847399 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:29.847408 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:29.847448 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:29.847838 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:29.847839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:29.847842 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:29.847890 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.847891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.848762 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.848763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.849890 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.849892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.850750 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.850751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.850837 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.850852 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 20:47:29.850921 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:47:29.850975 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":72,"threads_started":1}
I20260926 20:47:29.851066 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.847434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.0.98.65:37327)
0926 20:47:29.847647 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:29.847651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:29.847661 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:47:29.847700 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:29.847702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:29.847736 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:29.847962 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:29.848006 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.848679 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.849982 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.850658 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.850739 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.850836 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:29.850935 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 20:47:29.850976 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":96,"threads_started":1}
I20260926 20:47:29.851440 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.846963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:45495 (local address 127.0.0.1:40496)
0926 20:47:29.847359 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:29.847383 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:29.847453 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:29.847782 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:29.847785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:29.847788 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:29.847869 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.847872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.849261 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.849262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.850226 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.850227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.851154 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.851155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.851227 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.851232 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:47:29.851264 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 20:47:29.851310 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":10,"thread_start_us":79,"threads_started":1}
I20260926 20:47:29.851516 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.846963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:35705 (local address 127.0.0.1:53992)
0926 20:47:29.847163 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:29.847206 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:29.847270 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:29.847782 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:29.847785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:29.847790 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:29.847869 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.847872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.849293 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.849294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.850344 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.850347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.851186 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.851187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.851235 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.851237 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:47:29.851264 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 20:47:29.851310 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":6,"thread_start_us":78,"threads_started":1}
I20260926 20:47:29.851643 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.847153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.0.98.67:35705)
0926 20:47:29.847594 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:29.847598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:29.847607 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:47:29.847643 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:29.847645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:29.847662 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:29.847952 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:29.848029 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.849173 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.850427 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.851100 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.851366 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.851472 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:29.851499 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 20:47:29.851543 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":92,"threads_started":1}
I20260926 20:47:29.852073 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.847055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40496 (local address 127.0.98.66:45495)
0926 20:47:29.847273 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:29.847278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:29.847458 (+ 180us) server_negotiation.cc:407] Connection header received
0926 20:47:29.847534 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:29.847537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:29.847554 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:29.847953 (+ 399us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:29.848057 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.849174 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.850305 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.851043 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.851864 (+ 821us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.851954 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:29.851975 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 20:47:29.852009 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":106,"threads_started":1}
I20260926 20:47:29.852162 18035 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0ed53517db4fab90ec850022527f85 (DEFAULT_TABLE table=test-table [id=ebe5cc0480e14539bdedaf68624625ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:47:29.852574 18035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0ed53517db4fab90ec850022527f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:29.852725 18166 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0ed53517db4fab90ec850022527f85 (DEFAULT_TABLE table=test-table [id=ebe5cc0480e14539bdedaf68624625ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:47:29.853052 18166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0ed53517db4fab90ec850022527f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:29.856289 18378 tablet_bootstrap.cc:492] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845: Bootstrap starting.
I20260926 20:47:29.856561 18377 tablet_bootstrap.cc:492] T 9f0ed53517db4fab90ec850022527f85 P e79fb0d48dda4ed2a581298cba48aaa3: Bootstrap starting.
I20260926 20:47:29.856513 18297 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0ed53517db4fab90ec850022527f85 (DEFAULT_TABLE table=test-table [id=ebe5cc0480e14539bdedaf68624625ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:47:29.856840 18297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0ed53517db4fab90ec850022527f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:29.856971 18378 tablet_bootstrap.cc:654] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:29.857214 18377 tablet_bootstrap.cc:654] T 9f0ed53517db4fab90ec850022527f85 P e79fb0d48dda4ed2a581298cba48aaa3: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:29.857290 18378 log.cc:824] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:29.857540 18377 log.cc:824] T 9f0ed53517db4fab90ec850022527f85 P e79fb0d48dda4ed2a581298cba48aaa3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:29.858314 18377 tablet_bootstrap.cc:492] T 9f0ed53517db4fab90ec850022527f85 P e79fb0d48dda4ed2a581298cba48aaa3: No bootstrap required, opened a new log
I20260926 20:47:29.858347 18378 tablet_bootstrap.cc:492] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845: No bootstrap required, opened a new log
I20260926 20:47:29.858415 18378 ts_tablet_manager.cc:1397] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:47:29.858417 18377 ts_tablet_manager.cc:1397] T 9f0ed53517db4fab90ec850022527f85 P e79fb0d48dda4ed2a581298cba48aaa3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:47:29.860308 18381 tablet_bootstrap.cc:492] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd: Bootstrap starting.
I20260926 20:47:29.860778 18377 raft_consensus.cc:348] T 9f0ed53517db4fab90ec850022527f85 P e79fb0d48dda4ed2a581298cba48aaa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8b2efe9cbe4739872d90c0c5151845" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45495 } } peers { permanent_uuid: "0d6a9185440248859d01f2c9796f90dd" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35705 } } peers { permanent_uuid: "e79fb0d48dda4ed2a581298cba48aaa3" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 37327 } }
I20260926 20:47:29.860945 18377 raft_consensus.cc:374] T 9f0ed53517db4fab90ec850022527f85 P e79fb0d48dda4ed2a581298cba48aaa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:29.860880 18378 raft_consensus.cc:348] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8b2efe9cbe4739872d90c0c5151845" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45495 } } peers { permanent_uuid: "0d6a9185440248859d01f2c9796f90dd" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35705 } } peers { permanent_uuid: "e79fb0d48dda4ed2a581298cba48aaa3" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 37327 } }
I20260926 20:47:29.860991 18378 raft_consensus.cc:374] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:29.860987 18377 raft_consensus.cc:729] T 9f0ed53517db4fab90ec850022527f85 P e79fb0d48dda4ed2a581298cba48aaa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79fb0d48dda4ed2a581298cba48aaa3, State: Initialized, Role: FOLLOWER
I20260926 20:47:29.861011 18378 raft_consensus.cc:729] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f8b2efe9cbe4739872d90c0c5151845, State: Initialized, Role: FOLLOWER
I20260926 20:47:29.861048 18381 tablet_bootstrap.cc:654] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:29.861114 18377 consensus_queue.cc:260] T 9f0ed53517db4fab90ec850022527f85 P e79fb0d48dda4ed2a581298cba48aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8b2efe9cbe4739872d90c0c5151845" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45495 } } peers { permanent_uuid: "0d6a9185440248859d01f2c9796f90dd" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35705 } } peers { permanent_uuid: "e79fb0d48dda4ed2a581298cba48aaa3" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 37327 } }
I20260926 20:47:29.861126 18378 consensus_queue.cc:260] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8b2efe9cbe4739872d90c0c5151845" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45495 } } peers { permanent_uuid: "0d6a9185440248859d01f2c9796f90dd" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35705 } } peers { permanent_uuid: "e79fb0d48dda4ed2a581298cba48aaa3" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 37327 } }
I20260926 20:47:29.861400 18378 ts_tablet_manager.cc:1428] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:47:29.861394 18381 log.cc:824] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:29.861426 18377 ts_tablet_manager.cc:1428] T 9f0ed53517db4fab90ec850022527f85 P e79fb0d48dda4ed2a581298cba48aaa3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:47:29.861477 18232 heartbeater.cc:503] Master 127.0.98.126:34399 was elected leader, sending a full tablet report...
I20260926 20:47:29.861533 18101 heartbeater.cc:503] Master 127.0.98.126:34399 was elected leader, sending a full tablet report...
I20260926 20:47:29.862073 18381 tablet_bootstrap.cc:492] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd: No bootstrap required, opened a new log
I20260926 20:47:29.862140 18381 ts_tablet_manager.cc:1397] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:47:29.864148 18381 raft_consensus.cc:348] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8b2efe9cbe4739872d90c0c5151845" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45495 } } peers { permanent_uuid: "0d6a9185440248859d01f2c9796f90dd" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35705 } } peers { permanent_uuid: "e79fb0d48dda4ed2a581298cba48aaa3" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 37327 } }
I20260926 20:47:29.864267 18381 raft_consensus.cc:374] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:29.864288 18381 raft_consensus.cc:729] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d6a9185440248859d01f2c9796f90dd, State: Initialized, Role: FOLLOWER
I20260926 20:47:29.864449 18381 consensus_queue.cc:260] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8b2efe9cbe4739872d90c0c5151845" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45495 } } peers { permanent_uuid: "0d6a9185440248859d01f2c9796f90dd" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35705 } } peers { permanent_uuid: "e79fb0d48dda4ed2a581298cba48aaa3" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 37327 } }
I20260926 20:47:29.864675 18381 ts_tablet_manager.cc:1428] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:47:29.864744 18363 heartbeater.cc:503] Master 127.0.98.126:34399 was elected leader, sending a full tablet report...
I20260926 20:47:29.865391 18382 raft_consensus.cc:482] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:47:29.865490 18382 raft_consensus.cc:504] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8b2efe9cbe4739872d90c0c5151845" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45495 } } peers { permanent_uuid: "0d6a9185440248859d01f2c9796f90dd" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35705 } } peers { permanent_uuid: "e79fb0d48dda4ed2a581298cba48aaa3" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 37327 } }
I20260926 20:47:29.865759 18382 leader_election.cc:288] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d6a9185440248859d01f2c9796f90dd (127.0.98.67:35705), e79fb0d48dda4ed2a581298cba48aaa3 (127.0.98.65:37327)
I20260926 20:47:29.868683 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.865894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:35705 (local address 127.0.98.66:54099)
0926 20:47:29.865947 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:29.865960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:29.866024 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:29.866209 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:29.866210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:29.866212 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:29.866288 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.866290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.867041 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.867042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.867901 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.867904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.868435 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.868436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.868510 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.868513 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:47:29.868565 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 20:47:29.868606 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 20:47:29.868788 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.866092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:33197 (local address 127.0.98.65:37327)
0926 20:47:29.866132 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:29.866134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:29.866249 (+ 115us) server_negotiation.cc:407] Connection header received
0926 20:47:29.866295 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:29.866296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:29.866307 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:29.866359 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:29.866574 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.867112 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.868068 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.868437 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.868504 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.868512 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:29.868675 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 20:47:29.868719 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:47:29.868788 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.866018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:37327 (local address 127.0.98.66:33197)
0926 20:47:29.866185 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:29.866194 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:29.866247 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:29.866366 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:29.866367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:29.866369 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:29.866439 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.866440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.867189 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.867189 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.867987 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:29.867989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.868512 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.868513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:29.868602 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.868606 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:47:29.868662 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 20:47:29.868701 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":87,"threads_started":1}
I20260926 20:47:29.869194 18055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0ed53517db4fab90ec850022527f85" candidate_uuid: "6f8b2efe9cbe4739872d90c0c5151845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79fb0d48dda4ed2a581298cba48aaa3" is_pre_election: true
I20260926 20:47:29.869325 18055 raft_consensus.cc:2454] T 9f0ed53517db4fab90ec850022527f85 P e79fb0d48dda4ed2a581298cba48aaa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f8b2efe9cbe4739872d90c0c5151845 in term 0.
I20260926 20:47:29.869524 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.865982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:54099 (local address 127.0.98.67:35705)
0926 20:47:29.866084 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:29.866086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:29.866092 (+ 6us) server_negotiation.cc:407] Connection header received
0926 20:47:29.866115 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:29.866116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:29.866125 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:29.866177 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:29.866414 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.866979 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.867975 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.868365 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.868416 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.868424 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:29.869431 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0926 20:47:29.869468 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 20:47:29.869524 18119 leader_election.cc:302] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f8b2efe9cbe4739872d90c0c5151845, e79fb0d48dda4ed2a581298cba48aaa3; no voters:
I20260926 20:47:29.869637 18382 raft_consensus.cc:2781] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:47:29.869704 18382 raft_consensus.cc:482] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:47:29.869732 18382 raft_consensus.cc:3037] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:29.869755 18317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0ed53517db4fab90ec850022527f85" candidate_uuid: "6f8b2efe9cbe4739872d90c0c5151845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d6a9185440248859d01f2c9796f90dd" is_pre_election: true
I20260926 20:47:29.869869 18317 raft_consensus.cc:2454] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f8b2efe9cbe4739872d90c0c5151845 in term 0.
I20260926 20:47:29.870394 18382 raft_consensus.cc:504] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8b2efe9cbe4739872d90c0c5151845" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45495 } } peers { permanent_uuid: "0d6a9185440248859d01f2c9796f90dd" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35705 } } peers { permanent_uuid: "e79fb0d48dda4ed2a581298cba48aaa3" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 37327 } }
I20260926 20:47:29.870522 18382 leader_election.cc:288] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 [CANDIDATE]: Term 1 election: Requested vote from peers 0d6a9185440248859d01f2c9796f90dd (127.0.98.67:35705), e79fb0d48dda4ed2a581298cba48aaa3 (127.0.98.65:37327)
I20260926 20:47:29.870692 18317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0ed53517db4fab90ec850022527f85" candidate_uuid: "6f8b2efe9cbe4739872d90c0c5151845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d6a9185440248859d01f2c9796f90dd"
I20260926 20:47:29.870692 18055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0ed53517db4fab90ec850022527f85" candidate_uuid: "6f8b2efe9cbe4739872d90c0c5151845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79fb0d48dda4ed2a581298cba48aaa3"
I20260926 20:47:29.870761 18317 raft_consensus.cc:3037] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:29.870761 18055 raft_consensus.cc:3037] T 9f0ed53517db4fab90ec850022527f85 P e79fb0d48dda4ed2a581298cba48aaa3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:29.871567 18317 raft_consensus.cc:2454] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f8b2efe9cbe4739872d90c0c5151845 in term 1.
I20260926 20:47:29.871752 18119 leader_election.cc:302] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d6a9185440248859d01f2c9796f90dd, 6f8b2efe9cbe4739872d90c0c5151845; no voters:
I20260926 20:47:29.871764 18055 raft_consensus.cc:2454] T 9f0ed53517db4fab90ec850022527f85 P e79fb0d48dda4ed2a581298cba48aaa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f8b2efe9cbe4739872d90c0c5151845 in term 1.
I20260926 20:47:29.871840 18382 raft_consensus.cc:2781] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:29.871991 18382 raft_consensus.cc:686] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 [term 1 LEADER]: Becoming Leader. State: Replica: 6f8b2efe9cbe4739872d90c0c5151845, State: Running, Role: LEADER
I20260926 20:47:29.872094 18382 consensus_queue.cc:237] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8b2efe9cbe4739872d90c0c5151845" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45495 } } peers { permanent_uuid: "0d6a9185440248859d01f2c9796f90dd" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35705 } } peers { permanent_uuid: "e79fb0d48dda4ed2a581298cba48aaa3" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 37327 } }
I20260926 20:47:29.872768 17912 catalog_manager.cc:5495] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f8b2efe9cbe4739872d90c0c5151845 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "6f8b2efe9cbe4739872d90c0c5151845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8b2efe9cbe4739872d90c0c5151845" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d6a9185440248859d01f2c9796f90dd" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79fb0d48dda4ed2a581298cba48aaa3" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 37327 } health_report { overall_health: UNKNOWN } } }
I20260926 20:47:29.881742 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.879606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40508 (local address 127.0.98.66:45495)
0926 20:47:29.879662 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:29.879664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:29.879683 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:47:29.879741 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:29.879742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:29.879760 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:29.879816 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:47:29.879912 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.880346 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.881050 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:29.881236 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:29.881285 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:29.881463 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:47:29.881548 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:47:29.881599 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:29.881614 (+ 15us) server_negotiation.cc:299] Negotiation successful
0926 20:47:29.881627 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 20:47:29.883267 18055 raft_consensus.cc:1261] T 9f0ed53517db4fab90ec850022527f85 P e79fb0d48dda4ed2a581298cba48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer 6f8b2efe9cbe4739872d90c0c5151845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:47:29.883293 18317 raft_consensus.cc:1261] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd [term 1 FOLLOWER]: Refusing update from remote peer 6f8b2efe9cbe4739872d90c0c5151845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:47:29.883574 18382 consensus_queue.cc:1035] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79fb0d48dda4ed2a581298cba48aaa3" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 37327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:47:29.883737 18387 consensus_queue.cc:1035] T 9f0ed53517db4fab90ec850022527f85 P 6f8b2efe9cbe4739872d90c0c5151845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d6a9185440248859d01f2c9796f90dd" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:47:29.885968 18390 mvcc.cc:205] Tried to move back new op lower bound from 7333706341919748096 to 7333706341876662272. Current Snapshot: MvccSnapshot[applied={T|T < 7333706341919748096}]
I20260926 20:47:29.888103 17913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:47:29.888173 17913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:47:29.888873 17913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a0aa532caf045fdb9e8d7c516ddaad5: Sending DeleteTablet for 3 replicas of tablet 9f0ed53517db4fab90ec850022527f85
I20260926 20:47:29.889143 18166 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0ed53517db4fab90ec850022527f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:47:29 UTC) from {username='slave'} at 127.0.0.1:40496
I20260926 20:47:29.889230 18297 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0ed53517db4fab90ec850022527f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:47:29 UTC) from {username='slave'} at 127.0.0.1:53992
I20260926 20:47:29.889439 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 17973
I20260926 20:47:29.889489 18035 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0ed53517db4fab90ec850022527f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:47:29 UTC) from {username='slave'} at 127.0.0.1:36630
I20260926 20:47:29.890046 18401 tablet_replica.cc:330] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd: stopping tablet replica
I20260926 20:47:29.890177 18401 raft_consensus.cc:2229] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:47:29.890324 18401 raft_consensus.cc:2258] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:47:29.890820 18401 ts_tablet_manager.cc:1905] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:47:29.892570 18401 ts_tablet_manager.cc:1918] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:47:29.892644 18401 log.cc:1196] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/9f0ed53517db4fab90ec850022527f85
I20260926 20:47:29.892906 18401 ts_tablet_manager.cc:1939] T 9f0ed53517db4fab90ec850022527f85 P 0d6a9185440248859d01f2c9796f90dd: Deleting consensus metadata
I20260926 20:47:29.893301 17897 catalog_manager.cc:4841] TS 0d6a9185440248859d01f2c9796f90dd (127.0.98.67:35705): tablet 9f0ed53517db4fab90ec850022527f85 (table test-table [id=ebe5cc0480e14539bdedaf68624625ac]) successfully deleted
W20260926 20:47:29.895521 17897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:37327 (error 108)
I20260926 20:47:29.895557 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 18104
I20260926 20:47:29.896041 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.895928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:37327 (local address 127.0.0.1:36642)
0926 20:47:29.895975 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:29.896009 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 20:47:29.896150 17897 catalog_manager.cc:4584] TS e79fb0d48dda4ed2a581298cba48aaa3 (127.0.98.65:37327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f0ed53517db4fab90ec850022527f85: Network error: Client connection negotiation failed: client connection to 127.0.98.65:37327: connect: Connection refused (error 111)
I20260926 20:47:29.900735 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 18235
I20260926 20:47:29.900820 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:29.900703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:45495 (local address 127.0.0.1:40520)
0926 20:47:29.900752 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:29.900781 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.66:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 20:47:29.905287 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 17882
2026-09-26T20:47:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (648 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:29 dist-test-slave-4hd3 krb5kdc[18405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:29 dist-test-slave-4hd3 krb5kdc[18405](info): set up 2 sockets
Sep 26 20:47:29 dist-test-slave-4hd3 krb5kdc[18405](info): commencing operation
krb5kdc: starting...
W20260926 20:47:30.943480 393 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:31 dist-test-slave-4hd3 krb5kdc[18405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:47:31.058254 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:43397
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:43397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:31.137527 18418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:31.137770 18418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:31.137809 18418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:31.138957 18418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:31.139026 18418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:31.139050 18418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:31.139071 18418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:31.139099 18418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:31.140548 18418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:43397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:43397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:31.140813 18418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:31.141039 18418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:31.145798 18424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:31.145797 18423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:31.146005 18418 server_base.cc:1034] running on GCE node
W20260926 20:47:31.145802 18426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:31.146325 18418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:31.146618 18418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:31.147835 18418 hybrid_clock.cc:648] HybridClock initialized: now 1790455651147814 us; error 31 us; skew 500 ppm
Sep 26 20:47:31 dist-test-slave-4hd3 krb5kdc[18405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:31.150365 18418 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:47:31.150836 18418 webserver.cc:460] Webserver started at http://127.0.98.126:36805/ using document root <none> and password file <none>
I20260926 20:47:31.151051 18418 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:31.151116 18418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:31.151263 18418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:31.152371 18418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "26b5e3e1204140d39f91df89eb319305"
format_stamp: "Formatted at 2026-09-26 20:47:31 on dist-test-slave-4hd3"
server_key: "bfe70edb32828f08ab514927d5b65936"
server_key_iv: "055164f7d680917212e4eec34c79552d"
server_key_version: "encryptionkey@0"
I20260926 20:47:31.152717 18418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "26b5e3e1204140d39f91df89eb319305"
format_stamp: "Formatted at 2026-09-26 20:47:31 on dist-test-slave-4hd3"
server_key: "bfe70edb32828f08ab514927d5b65936"
server_key_iv: "055164f7d680917212e4eec34c79552d"
server_key_version: "encryptionkey@0"
I20260926 20:47:31.154032 18418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:47:31.154748 18433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:31.154966 18418 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:47:31.155073 18418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "26b5e3e1204140d39f91df89eb319305"
format_stamp: "Formatted at 2026-09-26 20:47:31 on dist-test-slave-4hd3"
server_key: "bfe70edb32828f08ab514927d5b65936"
server_key_iv: "055164f7d680917212e4eec34c79552d"
server_key_version: "encryptionkey@0"
I20260926 20:47:31.155146 18418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:31.165130 18418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:31.167418 18418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:31.167577 18418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:31.171634 18418 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:43397
I20260926 20:47:31.171665 18485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:43397 every 8 connection(s)
I20260926 20:47:31.172057 18418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:31.172588 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 18418
I20260926 20:47:31.172703 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:31.172905 393 external_mini_cluster.cc:1428] Setting key 95cd24f118a8a522817b630dff9c731c
I20260926 20:47:31.173135 18486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:31.176651 18486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305: Bootstrap starting.
I20260926 20:47:31.177287 18486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:31.177597 18486 log.cc:824] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:31.178505 18486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305: No bootstrap required, opened a new log
Sep 26 20:47:31 dist-test-slave-4hd3 krb5kdc[18405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:47:31.180748 18486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26b5e3e1204140d39f91df89eb319305" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43397 } }
I20260926 20:47:31.180891 18486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:31.180913 18486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b5e3e1204140d39f91df89eb319305, State: Initialized, Role: FOLLOWER
I20260926 20:47:31.181046 18486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26b5e3e1204140d39f91df89eb319305" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43397 } }
I20260926 20:47:31.181106 18486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:31.181128 18486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:31.181190 18486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:31.181739 18486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26b5e3e1204140d39f91df89eb319305" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43397 } }
I20260926 20:47:31.181854 18486 leader_election.cc:302] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26b5e3e1204140d39f91df89eb319305; no voters:
I20260926 20:47:31.182026 18486 leader_election.cc:288] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:31.182072 18491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:31.182250 18491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305 [term 1 LEADER]: Becoming Leader. State: Replica: 26b5e3e1204140d39f91df89eb319305, State: Running, Role: LEADER
I20260926 20:47:31.182305 18486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:31.182431 18491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26b5e3e1204140d39f91df89eb319305" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43397 } }
I20260926 20:47:31.182883 18491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26b5e3e1204140d39f91df89eb319305. Latest consensus state: current_term: 1 leader_uuid: "26b5e3e1204140d39f91df89eb319305" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26b5e3e1204140d39f91df89eb319305" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43397 } } }
I20260926 20:47:31.182893 18493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26b5e3e1204140d39f91df89eb319305" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26b5e3e1204140d39f91df89eb319305" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43397 } } }
I20260926 20:47:31.183086 18493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:31.183290 18491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:31.183578 18500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:31.184269 18500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:31.184716 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:31.173439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.0.98.126:43397)
0926 20:47:31.173617 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:31.173627 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:31.173658 (+ 31us) server_negotiation.cc:407] Connection header received
0926 20:47:31.174192 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:31.174201 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:31.174265 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:31.174420 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:31.175062 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.176451 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.177268 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.177883 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.178032 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:31.180756 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:31.180759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:31.180760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:31.180769 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:31.182897 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:31.183393 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:31.183396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:31.183397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:31.183500 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:31.183770 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:31.183771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:31.183771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:31.184085 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:31.184237 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:31.184399 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 20:47:31.184531 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260926 20:47:31.185770 18500 catalog_manager.cc:1347] Generated new cluster ID: 44a7a04420054498b815624d31075da0
I20260926 20:47:31.185824 18500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:47:31.203807 18500 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:31.204525 18500 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:31.216524 18500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305: Generated new TSK 0
I20260926 20:47:31.216717 18500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260926 20:47:31.250319 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:43397
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:31.331154 18512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:31.331387 18512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:31.331427 18512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:31.332616 18512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:31.332685 18512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:31.332731 18512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:47:31.334249 18512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:43397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:31.334512 18512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:31.334733 18512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:31.340157 18520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:31.340162 18518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:31.340217 18517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:31.340178 18512 server_base.cc:1034] running on GCE node
I20260926 20:47:31.340514 18512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:31.340761 18512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:31.341914 18512 hybrid_clock.cc:648] HybridClock initialized: now 1790455651341894 us; error 36 us; skew 500 ppm
Sep 26 20:47:31 dist-test-slave-4hd3 krb5kdc[18405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:31.344452 18512 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:47:31.344906 18512 webserver.cc:460] Webserver started at http://127.0.98.65:38777/ using document root <none> and password file <none>
I20260926 20:47:31.345153 18512 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:31.345220 18512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:31.345366 18512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:31.346410 18512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "07f58e07339441a1a7bda605e1b0caad"
format_stamp: "Formatted at 2026-09-26 20:47:31 on dist-test-slave-4hd3"
server_key: "2adf19b86cec3a79dc538a32fc9d9841"
server_key_iv: "2da45ac322fbe66d0772739765dcea63"
server_key_version: "encryptionkey@0"
I20260926 20:47:31.346768 18512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "07f58e07339441a1a7bda605e1b0caad"
format_stamp: "Formatted at 2026-09-26 20:47:31 on dist-test-slave-4hd3"
server_key: "2adf19b86cec3a79dc538a32fc9d9841"
server_key_iv: "2da45ac322fbe66d0772739765dcea63"
server_key_version: "encryptionkey@0"
I20260926 20:47:31.348143 18512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:47:31.348853 18527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:31.349009 18512 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:47:31.349105 18512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "07f58e07339441a1a7bda605e1b0caad"
format_stamp: "Formatted at 2026-09-26 20:47:31 on dist-test-slave-4hd3"
server_key: "2adf19b86cec3a79dc538a32fc9d9841"
server_key_iv: "2da45ac322fbe66d0772739765dcea63"
server_key_version: "encryptionkey@0"
I20260926 20:47:31.349175 18512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:31.360471 18512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:31.362454 18512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:31.362610 18512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:31.362820 18512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:31.363111 18512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:31.363160 18512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:31.363200 18512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:31.363235 18512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:31.369166 18512 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:38621
I20260926 20:47:31.369207 18640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:38621 every 8 connection(s)
I20260926 20:47:31.369557 18512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:47:31.375468 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 18512
I20260926 20:47:31.375557 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:47:31.375834 393 external_mini_cluster.cc:1428] Setting key 00f5339246c61053f679a018d6b7b26b
Sep 26 20:47:31 dist-test-slave-4hd3 krb5kdc[18405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:47:31.379887 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:31.370289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:36279 (local address 127.0.98.126:43397)
0926 20:47:31.370365 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:31.370367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:31.370552 (+ 185us) server_negotiation.cc:407] Connection header received
0926 20:47:31.371515 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:31.371516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:31.371526 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:31.371587 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:31.372265 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.372759 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.373832 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.374048 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.374106 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:31.376556 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:31.376558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:31.376558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:31.376564 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:31.378255 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:31.378780 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:31.378781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:31.378781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:31.378846 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:31.379160 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:31.379161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:31.379161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:31.379283 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:31.379380 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:31.379675 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 20:47:31.379820 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:47:31.380142 18643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:31.370260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:43397 (local address 127.0.98.65:36279)
0926 20:47:31.370423 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:31.370465 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:31.371414 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:31.371642 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:31.371646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:31.371986 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:31.372183 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:31.372187 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.372879 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.372880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:31.373747 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:31.373750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.374125 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.374126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:31.374238 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:31.374779 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:47:31.374787 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:31.376438 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:31.378367 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:31.378375 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:31.378378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:31.378710 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:31.378914 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:31.378915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:31.378915 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:31.379070 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:31.379357 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:31.379364 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:31.379534 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 20:47:31.379651 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 20:47:31.379860 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260926 20:47:31.380630 18641 heartbeater.cc:347] Connected to a master server at 127.0.98.126:43397
I20260926 20:47:31.380738 18641 heartbeater.cc:464] Registering TS with master...
I20260926 20:47:31.380915 18641 heartbeater.cc:511] Master 127.0.98.126:43397 requested a full tablet report, sending...
I20260926 20:47:31.381460 18450 ts_manager.cc:194] Registered new tserver with Master: 07f58e07339441a1a7bda605e1b0caad (127.0.98.65:38621)
I20260926 20:47:31.382057 18450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:36279
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260926 20:47:31.402161 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:43397
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:31.481113 18646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:31.481323 18646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:31.481379 18646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:31.482685 18646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:31.482729 18646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:31.482758 18646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:47:31.484402 18646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:43397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:31.484648 18646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:31.484875 18646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:31.489768 18654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:31.489745 18651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:31.489745 18652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:31.490002 18646 server_base.cc:1034] running on GCE node
I20260926 20:47:31.490237 18646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:31.490474 18646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:31.491631 18646 hybrid_clock.cc:648] HybridClock initialized: now 1790455651491616 us; error 27 us; skew 500 ppm
Sep 26 20:47:31 dist-test-slave-4hd3 krb5kdc[18405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:31.494131 18646 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:47:31.494585 18646 webserver.cc:460] Webserver started at http://127.0.98.66:34275/ using document root <none> and password file <none>
I20260926 20:47:31.494767 18646 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:31.494807 18646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:31.494902 18646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:31.495960 18646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "757e72ee7da54de1ac8e1541798dd8a0"
format_stamp: "Formatted at 2026-09-26 20:47:31 on dist-test-slave-4hd3"
server_key: "6f7a857c537f7291e863449e4d58aae1"
server_key_iv: "d4daf67190a303e6e1d2591a691762d3"
server_key_version: "encryptionkey@0"
I20260926 20:47:31.496306 18646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "757e72ee7da54de1ac8e1541798dd8a0"
format_stamp: "Formatted at 2026-09-26 20:47:31 on dist-test-slave-4hd3"
server_key: "6f7a857c537f7291e863449e4d58aae1"
server_key_iv: "d4daf67190a303e6e1d2591a691762d3"
server_key_version: "encryptionkey@0"
I20260926 20:47:31.497619 18646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:47:31.498399 18661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:31.498590 18646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:31.498677 18646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "757e72ee7da54de1ac8e1541798dd8a0"
format_stamp: "Formatted at 2026-09-26 20:47:31 on dist-test-slave-4hd3"
server_key: "6f7a857c537f7291e863449e4d58aae1"
server_key_iv: "d4daf67190a303e6e1d2591a691762d3"
server_key_version: "encryptionkey@0"
I20260926 20:47:31.498746 18646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:31.512275 18646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:31.514091 18646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:31.514240 18646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:31.514451 18646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:31.514753 18646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:31.514804 18646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:31.514838 18646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:31.514883 18646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:31.520715 18646 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:46301
I20260926 20:47:31.520764 18774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:46301 every 8 connection(s)
I20260926 20:47:31.521063 18646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:47:31.526671 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 18646
I20260926 20:47:31.526765 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
Sep 26 20:47:31 dist-test-slave-4hd3 krb5kdc[18405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:47:31.526978 393 external_mini_cluster.cc:1428] Setting key 4550af56795558bbc2496eb4677280cb
I20260926 20:47:31.529949 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:31.521707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:36399 (local address 127.0.98.126:43397)
0926 20:47:31.521754 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:31.521756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:31.521980 (+ 224us) server_negotiation.cc:407] Connection header received
0926 20:47:31.522851 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:31.522852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:31.522866 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:31.522918 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:31.523464 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.523976 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.524965 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.525155 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.525210 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:31.527313 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:31.527315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:31.527315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:31.527327 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:31.528627 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:31.529020 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:31.529021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:31.529022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:31.529076 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:31.529317 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:31.529318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:31.529318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:31.529432 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:31.529501 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:31.529774 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 20:47:31.529854 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:47:31.530135 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:31.521683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:43397 (local address 127.0.98.66:36399)
0926 20:47:31.521838 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:31.521892 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:31.522767 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:31.522978 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:31.522981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:31.523218 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:31.523399 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:31.523403 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.524068 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.524069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:31.524888 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:31.524890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.525245 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.525247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:31.525366 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:31.525787 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:47:31.525794 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:31.527212 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:31.528704 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:31.528711 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:31.528713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:31.528933 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:31.529140 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:31.529141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:31.529141 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:31.529252 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:31.529505 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:31.529511 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:31.529644 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 20:47:31.529738 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:47:31.529897 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260926 20:47:31.530624 18775 heartbeater.cc:347] Connected to a master server at 127.0.98.126:43397
I20260926 20:47:31.530725 18775 heartbeater.cc:464] Registering TS with master...
I20260926 20:47:31.530915 18775 heartbeater.cc:511] Master 127.0.98.126:43397 requested a full tablet report, sending...
I20260926 20:47:31.531299 18450 ts_manager.cc:194] Registered new tserver with Master: 757e72ee7da54de1ac8e1541798dd8a0 (127.0.98.66:46301)
I20260926 20:47:31.531679 18450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:36399
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260926 20:47:31.550540 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:43397
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:31.631325 18780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:31.631577 18780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:31.631613 18780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:31.632798 18780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:31.632869 18780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:31.632913 18780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:47:31.634418 18780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:43397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:31.634672 18780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:31.634897 18780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:31.639819 18788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:31.639837 18785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:31.639866 18786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:31.640141 18780 server_base.cc:1034] running on GCE node
I20260926 20:47:31.640354 18780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:31.640599 18780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:31.641772 18780 hybrid_clock.cc:648] HybridClock initialized: now 1790455651641761 us; error 35 us; skew 500 ppm
Sep 26 20:47:31 dist-test-slave-4hd3 krb5kdc[18405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:31.644238 18780 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:47:31.644682 18780 webserver.cc:460] Webserver started at http://127.0.98.67:32841/ using document root <none> and password file <none>
I20260926 20:47:31.644902 18780 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:31.644965 18780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:31.645107 18780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:31.646121 18780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "e03d1d3790f54375abb2b81c5bc9d678"
format_stamp: "Formatted at 2026-09-26 20:47:31 on dist-test-slave-4hd3"
server_key: "153cbeccd57e7f3303b7323f20cabc34"
server_key_iv: "2ae7be3c4f4ddade6b9680e0bc58d8f3"
server_key_version: "encryptionkey@0"
I20260926 20:47:31.646474 18780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "e03d1d3790f54375abb2b81c5bc9d678"
format_stamp: "Formatted at 2026-09-26 20:47:31 on dist-test-slave-4hd3"
server_key: "153cbeccd57e7f3303b7323f20cabc34"
server_key_iv: "2ae7be3c4f4ddade6b9680e0bc58d8f3"
server_key_version: "encryptionkey@0"
I20260926 20:47:31.647856 18780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:47:31.648656 18795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:31.648844 18780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:31.648947 18780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "e03d1d3790f54375abb2b81c5bc9d678"
format_stamp: "Formatted at 2026-09-26 20:47:31 on dist-test-slave-4hd3"
server_key: "153cbeccd57e7f3303b7323f20cabc34"
server_key_iv: "2ae7be3c4f4ddade6b9680e0bc58d8f3"
server_key_version: "encryptionkey@0"
I20260926 20:47:31.649029 18780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:31.666937 18780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:31.669188 18780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:31.669356 18780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:31.669596 18780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:31.669909 18780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:31.669960 18780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:31.670001 18780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:31.670029 18780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:31.675998 18780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:34257
I20260926 20:47:31.676038 18908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:34257 every 8 connection(s)
I20260926 20:47:31.676359 18780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:47:31 dist-test-slave-4hd3 krb5kdc[18405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:47:31.685425 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 18780
I20260926 20:47:31.685521 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:47:31.685678 393 external_mini_cluster.cc:1428] Setting key 3f1694e6ff545519299d18150ae0961e
I20260926 20:47:31.685940 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:31.677065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:39731 (local address 127.0.98.126:43397)
0926 20:47:31.677113 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:31.677115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:31.677292 (+ 177us) server_negotiation.cc:407] Connection header received
0926 20:47:31.678262 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:31.678263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:31.678276 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:31.678381 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:31.678986 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.679555 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.680598 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.680791 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.680859 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:31.683019 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:31.683020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:31.683021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:31.683028 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:31.684451 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:31.684917 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:31.684919 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:31.684919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:31.684982 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:31.685226 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:31.685227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:31.685228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:31.685340 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:31.685445 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:31.685761 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 20:47:31.685841 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:47:31.686146 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:31.677022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:43397 (local address 127.0.98.67:39731)
0926 20:47:31.677178 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:31.677219 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:31.678157 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:31.678445 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:31.678448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:31.678700 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:31.678900 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:31.678905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.679635 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.679636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:31.680503 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:31.680505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.680866 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.680867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:31.680981 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:31.681450 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:47:31.681457 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:31.682856 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:31.684535 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:31.684541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:31.684542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:31.684794 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:31.685049 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:31.685050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:31.685050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:31.685156 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:31.685450 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:31.685456 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:31.685623 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 20:47:31.685741 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 20:47:31.685918 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260926 20:47:31.686653 18909 heartbeater.cc:347] Connected to a master server at 127.0.98.126:43397
I20260926 20:47:31.686733 18909 heartbeater.cc:464] Registering TS with master...
I20260926 20:47:31.686872 18909 heartbeater.cc:511] Master 127.0.98.126:43397 requested a full tablet report, sending...
I20260926 20:47:31.687182 18450 ts_manager.cc:194] Registered new tserver with Master: e03d1d3790f54375abb2b81c5bc9d678 (127.0.98.67:34257)
I20260926 20:47:31.687577 18450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:39731
I20260926 20:47:31.697407 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:47:31.705058 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:31.698757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.0.98.126:43397)
0926 20:47:31.698841 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:31.698843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:31.698937 (+ 94us) server_negotiation.cc:407] Connection header received
0926 20:47:31.699006 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:31.699007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:31.699017 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:31.699081 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:31.699645 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.700175 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.700911 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.701104 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.701183 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:31.702596 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:31.702597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:31.702598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:31.702604 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:31.703857 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:31.704228 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:31.704229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:31.704229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:31.704281 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:31.704504 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:31.704505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:31.704505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:31.704640 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:31.704746 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:31.704890 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 20:47:31.704950 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:47:31.706668 18450 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:47:31.708384 18450 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:47:31.714478 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:31.710409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:38621 (local address 127.0.0.1:41726)
0926 20:47:31.710846 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:31.710871 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:31.710906 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:31.711135 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:31.711136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:31.711139 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:31.711192 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:31.711194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.712225 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.712226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:31.713274 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:31.713277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.714181 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.714181 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:31.714256 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:31.714270 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 20:47:31.714352 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:47:31.714403 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":43,"thread_start_us":47,"threads_started":1}
I20260926 20:47:31.714509 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:31.710794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41726 (local address 127.0.98.65:38621)
0926 20:47:31.710990 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:31.710995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:31.711003 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:47:31.711032 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:31.711033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:31.711044 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:31.711128 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:31.711277 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.712134 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.713383 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.714097 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.714175 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:31.714239 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:31.714380 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 20:47:31.714429 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":102,"threads_started":1}
I20260926 20:47:31.714680 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:31.710399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46301 (local address 127.0.0.1:52444)
0926 20:47:31.710641 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:31.710682 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:31.710751 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:31.711044 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:31.711046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:31.711049 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:31.711101 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:31.711102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.712270 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.712271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:31.713434 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:31.713436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.714317 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.714318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:31.714389 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:31.714393 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:47:31.714442 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 20:47:31.714478 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":24,"thread_start_us":58,"threads_started":1}
I20260926 20:47:31.714843 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:31.710519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52444 (local address 127.0.98.66:46301)
0926 20:47:31.710696 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:31.710701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:31.710739 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:47:31.710841 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:31.710843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:31.710861 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:31.710946 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:31.711203 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.712167 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.713508 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.714180 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.714237 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:31.714283 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:31.714519 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 20:47:31.714696 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":94,"threads_started":1}
I20260926 20:47:31.715169 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:31.710404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:34257 (local address 127.0.0.1:41424)
0926 20:47:31.710684 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:31.710692 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:31.710751 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:31.711963 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:31.711964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:31.711966 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:31.712016 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:31.712017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.713269 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.713270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:31.714040 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:31.714042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.714942 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.714943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:31.715012 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:31.715016 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:47:31.715070 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:47:31.715106 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":73,"threads_started":1}
I20260926 20:47:31.715189 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:31.710618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.0.98.67:34257)
0926 20:47:31.711797 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:31.711802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:31.711811 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:47:31.711858 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:31.711860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:31.711874 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:31.711952 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:31.712074 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.713156 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.714117 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.714848 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.714939 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:31.714991 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:31.715084 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 20:47:31.715117 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":76,"threads_started":1}
I20260926 20:47:31.715557 18709 tablet_service.cc:1467] Processing CreateTablet for tablet 918c5dd7bdd547e7b482a110598b8864 (DEFAULT_TABLE table=test-table [id=8daeae6674164ac3afb640012a882544]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:47:31.715557 18575 tablet_service.cc:1467] Processing CreateTablet for tablet 918c5dd7bdd547e7b482a110598b8864 (DEFAULT_TABLE table=test-table [id=8daeae6674164ac3afb640012a882544]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:47:31.715919 18575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 918c5dd7bdd547e7b482a110598b8864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:31.715919 18709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 918c5dd7bdd547e7b482a110598b8864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:31.716307 18843 tablet_service.cc:1467] Processing CreateTablet for tablet 918c5dd7bdd547e7b482a110598b8864 (DEFAULT_TABLE table=test-table [id=8daeae6674164ac3afb640012a882544]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:47:31.716610 18843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 918c5dd7bdd547e7b482a110598b8864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:31.719040 18923 tablet_bootstrap.cc:492] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678: Bootstrap starting.
I20260926 20:47:31.719709 18923 tablet_bootstrap.cc:654] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:31.719744 18924 tablet_bootstrap.cc:492] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad: Bootstrap starting.
I20260926 20:47:31.719866 18925 tablet_bootstrap.cc:492] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0: Bootstrap starting.
I20260926 20:47:31.720258 18923 log.cc:824] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:31.720405 18924 tablet_bootstrap.cc:654] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:31.720417 18925 tablet_bootstrap.cc:654] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:31.720968 18923 tablet_bootstrap.cc:492] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678: No bootstrap required, opened a new log
I20260926 20:47:31.721026 18924 log.cc:824] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:31.721026 18925 log.cc:824] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:31.721045 18923 ts_tablet_manager.cc:1397] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:47:31.721751 18925 tablet_bootstrap.cc:492] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0: No bootstrap required, opened a new log
I20260926 20:47:31.721820 18925 ts_tablet_manager.cc:1397] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:47:31.721992 18924 tablet_bootstrap.cc:492] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad: No bootstrap required, opened a new log
I20260926 20:47:31.722062 18924 ts_tablet_manager.cc:1397] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:47:31.722936 18923 raft_consensus.cc:348] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f58e07339441a1a7bda605e1b0caad" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38621 } } peers { permanent_uuid: "757e72ee7da54de1ac8e1541798dd8a0" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46301 } } peers { permanent_uuid: "e03d1d3790f54375abb2b81c5bc9d678" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34257 } }
I20260926 20:47:31.723093 18923 raft_consensus.cc:374] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:31.723136 18923 raft_consensus.cc:729] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e03d1d3790f54375abb2b81c5bc9d678, State: Initialized, Role: FOLLOWER
I20260926 20:47:31.723255 18923 consensus_queue.cc:260] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f58e07339441a1a7bda605e1b0caad" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38621 } } peers { permanent_uuid: "757e72ee7da54de1ac8e1541798dd8a0" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46301 } } peers { permanent_uuid: "e03d1d3790f54375abb2b81c5bc9d678" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34257 } }
I20260926 20:47:31.723531 18923 ts_tablet_manager.cc:1428] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:47:31.723650 18909 heartbeater.cc:503] Master 127.0.98.126:43397 was elected leader, sending a full tablet report...
I20260926 20:47:31.724042 18925 raft_consensus.cc:348] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f58e07339441a1a7bda605e1b0caad" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38621 } } peers { permanent_uuid: "757e72ee7da54de1ac8e1541798dd8a0" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46301 } } peers { permanent_uuid: "e03d1d3790f54375abb2b81c5bc9d678" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34257 } }
I20260926 20:47:31.724042 18924 raft_consensus.cc:348] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f58e07339441a1a7bda605e1b0caad" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38621 } } peers { permanent_uuid: "757e72ee7da54de1ac8e1541798dd8a0" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46301 } } peers { permanent_uuid: "e03d1d3790f54375abb2b81c5bc9d678" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34257 } }
I20260926 20:47:31.724216 18924 raft_consensus.cc:374] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:31.724216 18925 raft_consensus.cc:374] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:31.724252 18924 raft_consensus.cc:729] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07f58e07339441a1a7bda605e1b0caad, State: Initialized, Role: FOLLOWER
I20260926 20:47:31.724252 18925 raft_consensus.cc:729] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757e72ee7da54de1ac8e1541798dd8a0, State: Initialized, Role: FOLLOWER
I20260926 20:47:31.724370 18924 consensus_queue.cc:260] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f58e07339441a1a7bda605e1b0caad" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38621 } } peers { permanent_uuid: "757e72ee7da54de1ac8e1541798dd8a0" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46301 } } peers { permanent_uuid: "e03d1d3790f54375abb2b81c5bc9d678" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34257 } }
I20260926 20:47:31.724364 18925 consensus_queue.cc:260] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f58e07339441a1a7bda605e1b0caad" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38621 } } peers { permanent_uuid: "757e72ee7da54de1ac8e1541798dd8a0" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46301 } } peers { permanent_uuid: "e03d1d3790f54375abb2b81c5bc9d678" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34257 } }
I20260926 20:47:31.724624 18641 heartbeater.cc:503] Master 127.0.98.126:43397 was elected leader, sending a full tablet report...
I20260926 20:47:31.724627 18925 ts_tablet_manager.cc:1428] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 20:47:31.724750 18775 heartbeater.cc:503] Master 127.0.98.126:43397 was elected leader, sending a full tablet report...
I20260926 20:47:31.724917 18924 ts_tablet_manager.cc:1428] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260926 20:47:31.771639 18776 tablet.cc:2369] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:47:31.870294 18642 tablet.cc:2369] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:47:31.926901 18910 tablet.cc:2369] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:47:31.970512 18930 raft_consensus.cc:482] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:47:31.970700 18930 raft_consensus.cc:504] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f58e07339441a1a7bda605e1b0caad" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38621 } } peers { permanent_uuid: "757e72ee7da54de1ac8e1541798dd8a0" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46301 } } peers { permanent_uuid: "e03d1d3790f54375abb2b81c5bc9d678" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34257 } }
I20260926 20:47:31.971099 18930 leader_election.cc:288] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 757e72ee7da54de1ac8e1541798dd8a0 (127.0.98.66:46301), e03d1d3790f54375abb2b81c5bc9d678 (127.0.98.67:34257)
I20260926 20:47:31.975345 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:31.971456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:39623 (local address 127.0.98.66:46301)
0926 20:47:31.971500 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:31.971501 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:31.971644 (+ 143us) server_negotiation.cc:407] Connection header received
0926 20:47:31.971685 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:31.971686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:31.971702 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:31.971798 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:31.972051 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.972792 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.974375 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.974971 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.975043 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:31.975054 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:31.975232 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 20:47:31.975272 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:47:31.975345 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:31.971436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:40617 (local address 127.0.98.67:34257)
0926 20:47:31.971488 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:31.971491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:31.971644 (+ 153us) server_negotiation.cc:407] Connection header received
0926 20:47:31.971685 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:31.971687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:31.971699 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:31.971798 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:31.972045 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.972792 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.974375 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.974962 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.975043 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:31.975054 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:31.975232 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 20:47:31.975272 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:47:31.975409 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:31.971338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46301 (local address 127.0.98.65:39623)
0926 20:47:31.971523 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:31.971536 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:31.971610 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:31.971817 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:31.971818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:31.971820 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:31.971964 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:31.971966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.972885 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.972886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:31.974266 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:31.974270 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.975060 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.975061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:31.975149 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:31.975152 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:47:31.975214 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 20:47:31.975270 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":92,"thread_start_us":111,"threads_started":1}
I20260926 20:47:31.975447 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:31.971338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:34257 (local address 127.0.98.65:40617)
0926 20:47:31.971523 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:31.971536 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:31.971610 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:31.971817 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:31.971818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:31.971821 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:31.971931 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:31.971935 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.972885 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.972886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:31.974267 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:31.974270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.975059 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:31.975060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:31.975148 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:31.975153 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:47:31.975214 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 20:47:31.975270 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":122,"thread_start_us":98,"threads_started":1}
I20260926 20:47:31.975842 18729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918c5dd7bdd547e7b482a110598b8864" candidate_uuid: "07f58e07339441a1a7bda605e1b0caad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757e72ee7da54de1ac8e1541798dd8a0" is_pre_election: true
I20260926 20:47:31.975903 18863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918c5dd7bdd547e7b482a110598b8864" candidate_uuid: "07f58e07339441a1a7bda605e1b0caad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e03d1d3790f54375abb2b81c5bc9d678" is_pre_election: true
I20260926 20:47:31.976032 18863 raft_consensus.cc:2454] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07f58e07339441a1a7bda605e1b0caad in term 0.
I20260926 20:47:31.976032 18729 raft_consensus.cc:2454] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07f58e07339441a1a7bda605e1b0caad in term 0.
I20260926 20:47:31.976289 18529 leader_election.cc:302] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07f58e07339441a1a7bda605e1b0caad, e03d1d3790f54375abb2b81c5bc9d678; no voters:
I20260926 20:47:31.976436 18930 raft_consensus.cc:2781] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:47:31.976493 18930 raft_consensus.cc:482] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:47:31.976522 18930 raft_consensus.cc:3037] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:31.977224 18930 raft_consensus.cc:504] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f58e07339441a1a7bda605e1b0caad" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38621 } } peers { permanent_uuid: "757e72ee7da54de1ac8e1541798dd8a0" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46301 } } peers { permanent_uuid: "e03d1d3790f54375abb2b81c5bc9d678" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34257 } }
I20260926 20:47:31.977427 18930 leader_election.cc:288] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad [CANDIDATE]: Term 1 election: Requested vote from peers 757e72ee7da54de1ac8e1541798dd8a0 (127.0.98.66:46301), e03d1d3790f54375abb2b81c5bc9d678 (127.0.98.67:34257)
I20260926 20:47:31.977494 18729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918c5dd7bdd547e7b482a110598b8864" candidate_uuid: "07f58e07339441a1a7bda605e1b0caad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757e72ee7da54de1ac8e1541798dd8a0"
I20260926 20:47:31.977532 18863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918c5dd7bdd547e7b482a110598b8864" candidate_uuid: "07f58e07339441a1a7bda605e1b0caad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e03d1d3790f54375abb2b81c5bc9d678"
I20260926 20:47:31.977577 18729 raft_consensus.cc:3037] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:31.977591 18863 raft_consensus.cc:3037] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:31.978499 18729 raft_consensus.cc:2454] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07f58e07339441a1a7bda605e1b0caad in term 1.
I20260926 20:47:31.978499 18863 raft_consensus.cc:2454] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07f58e07339441a1a7bda605e1b0caad in term 1.
I20260926 20:47:31.978715 18530 leader_election.cc:302] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07f58e07339441a1a7bda605e1b0caad, 757e72ee7da54de1ac8e1541798dd8a0; no voters:
I20260926 20:47:31.978835 18930 raft_consensus.cc:2781] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:31.978983 18930 raft_consensus.cc:686] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad [term 1 LEADER]: Becoming Leader. State: Replica: 07f58e07339441a1a7bda605e1b0caad, State: Running, Role: LEADER
I20260926 20:47:31.979082 18930 consensus_queue.cc:237] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f58e07339441a1a7bda605e1b0caad" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38621 } } peers { permanent_uuid: "757e72ee7da54de1ac8e1541798dd8a0" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46301 } } peers { permanent_uuid: "e03d1d3790f54375abb2b81c5bc9d678" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34257 } }
I20260926 20:47:31.979926 18449 catalog_manager.cc:5495] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad reported cstate change: term changed from 0 to 1, leader changed from <none> to 07f58e07339441a1a7bda605e1b0caad (127.0.98.65). New cstate: current_term: 1 leader_uuid: "07f58e07339441a1a7bda605e1b0caad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f58e07339441a1a7bda605e1b0caad" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "757e72ee7da54de1ac8e1541798dd8a0" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e03d1d3790f54375abb2b81c5bc9d678" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34257 } health_report { overall_health: UNKNOWN } } }
I20260926 20:47:32.001036 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:31.998757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.0.98.65:38621)
0926 20:47:31.998826 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:31.998828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:31.998837 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:47:31.998874 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:31.998875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:31.998895 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:31.998957 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:47:31.999086 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:31.999579 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.000324 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.000535 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.000585 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.000724 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:47:32.000809 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:47:32.000881 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:32.000926 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 20:47:32.000936 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:47:32.002887 18729 raft_consensus.cc:1261] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0 [term 1 FOLLOWER]: Refusing update from remote peer 07f58e07339441a1a7bda605e1b0caad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:47:32.002902 18863 raft_consensus.cc:1261] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678 [term 1 FOLLOWER]: Refusing update from remote peer 07f58e07339441a1a7bda605e1b0caad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:47:32.003257 18930 consensus_queue.cc:1035] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad [LEADER]: Connected to new peer: Peer: permanent_uuid: "757e72ee7da54de1ac8e1541798dd8a0" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:47:32.003405 18934 consensus_queue.cc:1035] T 918c5dd7bdd547e7b482a110598b8864 P 07f58e07339441a1a7bda605e1b0caad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03d1d3790f54375abb2b81c5bc9d678" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:47:32.006242 18938 mvcc.cc:205] Tried to move back new op lower bound from 7333706350601199616 to 7333706350506917888. Current Snapshot: MvccSnapshot[applied={T|T < 7333706350601199616}]
I20260926 20:47:32.006342 18939 mvcc.cc:205] Tried to move back new op lower bound from 7333706350601199616 to 7333706350506917888. Current Snapshot: MvccSnapshot[applied={T|T < 7333706350601199616}]
I20260926 20:47:32.007936 18450 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:47:32.008026 18450 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:47:32.008792 18450 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26b5e3e1204140d39f91df89eb319305: Sending DeleteTablet for 3 replicas of tablet 918c5dd7bdd547e7b482a110598b8864
I20260926 20:47:32.009178 18709 tablet_service.cc:1514] Processing DeleteTablet for tablet 918c5dd7bdd547e7b482a110598b8864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:47:32 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:52444
I20260926 20:47:32.009171 18843 tablet_service.cc:1514] Processing DeleteTablet for tablet 918c5dd7bdd547e7b482a110598b8864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:47:32 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:41424
I20260926 20:47:32.009408 18947 tablet_replica.cc:330] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678: stopping tablet replica
I20260926 20:47:32.009622 18575 tablet_service.cc:1514] Processing DeleteTablet for tablet 918c5dd7bdd547e7b482a110598b8864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:47:32 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:41726
I20260926 20:47:32.009661 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 18512
I20260926 20:47:32.009532 18947 raft_consensus.cc:2229] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:47:32.010231 18947 raft_consensus.cc:2258] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:47:32.010502 18948 tablet_replica.cc:330] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0: stopping tablet replica
I20260926 20:47:32.010584 18948 raft_consensus.cc:2229] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:47:32.010651 18948 raft_consensus.cc:2258] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:47:32.010766 18947 ts_tablet_manager.cc:1905] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:47:32.010993 18948 ts_tablet_manager.cc:1905] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:47:32.012565 18948 ts_tablet_manager.cc:1918] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:47:32.012673 18948 log.cc:1196] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/918c5dd7bdd547e7b482a110598b8864
I20260926 20:47:32.012944 18948 ts_tablet_manager.cc:1939] T 918c5dd7bdd547e7b482a110598b8864 P 757e72ee7da54de1ac8e1541798dd8a0: Deleting consensus metadata
I20260926 20:47:32.012990 18947 ts_tablet_manager.cc:1918] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:47:32.013095 18947 log.cc:1196] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/918c5dd7bdd547e7b482a110598b8864
I20260926 20:47:32.013468 18947 ts_tablet_manager.cc:1939] T 918c5dd7bdd547e7b482a110598b8864 P e03d1d3790f54375abb2b81c5bc9d678: Deleting consensus metadata
I20260926 20:47:32.013456 18436 catalog_manager.cc:4841] TS 757e72ee7da54de1ac8e1541798dd8a0 (127.0.98.66:46301): tablet 918c5dd7bdd547e7b482a110598b8864 (table test-table [id=8daeae6674164ac3afb640012a882544]) successfully deleted
I20260926 20:47:32.013849 18435 catalog_manager.cc:4841] TS e03d1d3790f54375abb2b81c5bc9d678 (127.0.98.67:34257): tablet 918c5dd7bdd547e7b482a110598b8864 (table test-table [id=8daeae6674164ac3afb640012a882544]) successfully deleted
W20260926 20:47:32.016083 18434 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:38621 (error 108)
I20260926 20:47:32.016276 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 18646
I20260926 20:47:32.017077 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.016390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:38621 (local address 127.0.0.1:41732)
0926 20:47:32.017005 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:32.017051 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:38621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592}
W20260926 20:47:32.017212 18434 catalog_manager.cc:4584] TS 07f58e07339441a1a7bda605e1b0caad (127.0.98.65:38621): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 918c5dd7bdd547e7b482a110598b8864: Network error: Client connection negotiation failed: client connection to 127.0.98.65:38621: connect: Connection refused (error 111)
I20260926 20:47:32.022176 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 18780
I20260926 20:47:32.026778 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 18418
2026-09-26T20:47:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2121 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T20:47:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:32Z Disabled control of system clock
I20260926 20:47:32.054250 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34117
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:43541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:34117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:47:32.136019 18953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:32.136232 18953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:32.136255 18953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:32.137356 18953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:32.137391 18953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:32.137403 18953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:32.137416 18953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:32.137428 18953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:32.138698 18953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:34117
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:32.138923 18953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:32.139191 18953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:32.144026 18959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:32.144022 18958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:32.144161 18953 server_base.cc:1034] running on GCE node
W20260926 20:47:32.144021 18961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:32.144486 18953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:32.144773 18953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:32.145958 18953 hybrid_clock.cc:648] HybridClock initialized: now 1790455652145951 us; error 43 us; skew 500 ppm
I20260926 20:47:32.147174 18953 webserver.cc:460] Webserver started at http://127.0.98.126:46535/ using document root <none> and password file <none>
I20260926 20:47:32.147444 18953 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:32.147518 18953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:32.147658 18953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:32.148754 18953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "a7f58d82bbe143d0aea72319707a8974"
format_stamp: "Formatted at 2026-09-26 20:47:32 on dist-test-slave-4hd3"
server_key: "baf6f36bda3c88a6f3a9cef1bf6f0ae0"
server_key_iv: "d27f6a8838b5be1be7daeec4c75ec80c"
server_key_version: "encryptionkey@0"
I20260926 20:47:32.149118 18953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "a7f58d82bbe143d0aea72319707a8974"
format_stamp: "Formatted at 2026-09-26 20:47:32 on dist-test-slave-4hd3"
server_key: "baf6f36bda3c88a6f3a9cef1bf6f0ae0"
server_key_iv: "d27f6a8838b5be1be7daeec4c75ec80c"
server_key_version: "encryptionkey@0"
I20260926 20:47:32.150424 18953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:47:32.151150 18967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:32.151310 18953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:47:32.151409 18953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "a7f58d82bbe143d0aea72319707a8974"
format_stamp: "Formatted at 2026-09-26 20:47:32 on dist-test-slave-4hd3"
server_key: "baf6f36bda3c88a6f3a9cef1bf6f0ae0"
server_key_iv: "d27f6a8838b5be1be7daeec4c75ec80c"
server_key_version: "encryptionkey@0"
I20260926 20:47:32.151486 18953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:32.178918 18953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:32.179260 18953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:32.179416 18953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:32.183305 18953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:34117
I20260926 20:47:32.183388 19019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:34117 every 8 connection(s)
I20260926 20:47:32.183708 18953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:32.184690 19020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:32.186937 19020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974: Bootstrap starting.
I20260926 20:47:32.187506 19020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:32.187852 19020 log.cc:824] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:32.188573 19020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974: No bootstrap required, opened a new log
I20260926 20:47:32.189760 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 18953
I20260926 20:47:32.189863 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:32.190086 393 external_mini_cluster.cc:1428] Setting key 90dcd941f016a28cd983e4db954520ca
I20260926 20:47:32.190599 19020 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f58d82bbe143d0aea72319707a8974" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34117 } }
I20260926 20:47:32.190785 19020 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:32.190814 19020 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7f58d82bbe143d0aea72319707a8974, State: Initialized, Role: FOLLOWER
I20260926 20:47:32.190954 19020 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f58d82bbe143d0aea72319707a8974" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34117 } }
I20260926 20:47:32.191025 19020 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:32.191046 19020 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:32.191108 19020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:32.191745 19020 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f58d82bbe143d0aea72319707a8974" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34117 } }
I20260926 20:47:32.191867 19020 leader_election.cc:302] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7f58d82bbe143d0aea72319707a8974; no voters:
I20260926 20:47:32.192045 19020 leader_election.cc:288] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:32.192093 19025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:32.192232 19025 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974 [term 1 LEADER]: Becoming Leader. State: Replica: a7f58d82bbe143d0aea72319707a8974, State: Running, Role: LEADER
I20260926 20:47:32.192304 19020 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:32.192449 19025 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f58d82bbe143d0aea72319707a8974" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34117 } }
I20260926 20:47:32.192934 19027 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7f58d82bbe143d0aea72319707a8974. Latest consensus state: current_term: 1 leader_uuid: "a7f58d82bbe143d0aea72319707a8974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f58d82bbe143d0aea72319707a8974" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34117 } } }
I20260926 20:47:32.193040 19027 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:32.192938 19026 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7f58d82bbe143d0aea72319707a8974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f58d82bbe143d0aea72319707a8974" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34117 } } }
I20260926 20:47:32.193197 19026 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:32.193557 19034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:32.194011 19034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:32.195979 19034 catalog_manager.cc:1347] Generated new cluster ID: e12ddf9c4b114611865b7e3c4d427f02
I20260926 20:47:32.196030 19034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:47:32.196416 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.190587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.0.98.126:34117)
0926 20:47:32.190806 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:32.190814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:32.190847 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:47:32.190988 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:32.190994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:32.191074 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:32.191216 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:32.191509 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.193233 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.194334 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.194911 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.195044 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.195685 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:32.195689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:32.195689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:32.195920 (+ 231us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:32.195961 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:32.195976 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:32.196040 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:32.196157 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 20:47:32.196235 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260926 20:47:32.216645 19034 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:32.217340 19034 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:32.231251 19034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974: Generated new TSK 0
I20260926 20:47:32.231456 19034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:32.238282 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34117
--builtin_ntp_servers=127.0.98.84:43541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:47:32.319056 19044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:32.319259 19044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:32.319294 19044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:32.320475 19044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:32.320533 19044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:32.320574 19044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:47:32.321924 19044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:32.322165 19044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:32.322458 19044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:32.327299 19052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:32.327302 19049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:32.327277 19050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:32.327589 19044 server_base.cc:1034] running on GCE node
I20260926 20:47:32.327848 19044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:32.328094 19044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:32.329249 19044 hybrid_clock.cc:648] HybridClock initialized: now 1790455652329241 us; error 38 us; skew 500 ppm
I20260926 20:47:32.330493 19044 webserver.cc:460] Webserver started at http://127.0.98.65:41201/ using document root <none> and password file <none>
I20260926 20:47:32.330740 19044 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:32.330806 19044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:32.330936 19044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:32.332041 19044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "4da8afabe86e45a4a5cb8c7919aa721d"
format_stamp: "Formatted at 2026-09-26 20:47:32 on dist-test-slave-4hd3"
server_key: "1d4c5d1ba60b550a19b1b0b3f6a6341d"
server_key_iv: "5fa7ea2573f8e92ea9dc0f342bd26540"
server_key_version: "encryptionkey@0"
I20260926 20:47:32.332397 19044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "4da8afabe86e45a4a5cb8c7919aa721d"
format_stamp: "Formatted at 2026-09-26 20:47:32 on dist-test-slave-4hd3"
server_key: "1d4c5d1ba60b550a19b1b0b3f6a6341d"
server_key_iv: "5fa7ea2573f8e92ea9dc0f342bd26540"
server_key_version: "encryptionkey@0"
I20260926 20:47:32.333714 19044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:32.334417 19058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:32.334620 19044 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:47:32.334712 19044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "4da8afabe86e45a4a5cb8c7919aa721d"
format_stamp: "Formatted at 2026-09-26 20:47:32 on dist-test-slave-4hd3"
server_key: "1d4c5d1ba60b550a19b1b0b3f6a6341d"
server_key_iv: "5fa7ea2573f8e92ea9dc0f342bd26540"
server_key_version: "encryptionkey@0"
I20260926 20:47:32.334792 19044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:32.351243 19044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:32.351542 19044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:32.351687 19044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:32.351989 19044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:32.352305 19044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:32.352356 19044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:32.352402 19044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:32.352432 19044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:32.358191 19044 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:41921
I20260926 20:47:32.358238 19171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:41921 every 8 connection(s)
I20260926 20:47:32.358566 19044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:47:32.363178 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 19044
I20260926 20:47:32.363276 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:47:32.363488 393 external_mini_cluster.cc:1428] Setting key 376677318c217f20339b9a99dc8c1e37
I20260926 20:47:32.364442 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34117
--builtin_ntp_servers=127.0.98.84:43541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:47:32.366876 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.359247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:60995 (local address 127.0.98.126:34117)
0926 20:47:32.359305 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:32.359308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:32.359617 (+ 309us) server_negotiation.cc:407] Connection header received
0926 20:47:32.360178 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:32.360179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:32.360191 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:32.360245 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:32.360888 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.361379 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.362393 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.362584 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.362644 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.366527 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:32.366529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:32.366530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:32.366535 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:32.366553 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:32.366558 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:32.366614 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:32.366703 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 20:47:32.366741 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 20:47:32.366968 19174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.359253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34117 (local address 127.0.98.65:60995)
0926 20:47:32.359493 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:32.359528 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:32.360088 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:32.360317 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:32.360320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:32.360328 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:32.360816 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:32.360820 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.361478 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.361479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:32.362292 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:32.362294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.362661 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.362661 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:32.362767 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.366367 (+ 3600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:47:32.366368 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:32.366406 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:47:32.366435 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:32.366630 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:32.366635 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:32.366638 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:47:32.366689 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 20:47:32.366778 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":122,"threads_started":1}
I20260926 20:47:32.367446 19172 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34117
I20260926 20:47:32.367552 19172 heartbeater.cc:464] Registering TS with master...
I20260926 20:47:32.367813 19172 heartbeater.cc:511] Master 127.0.98.126:34117 requested a full tablet report, sending...
I20260926 20:47:32.368320 18984 ts_manager.cc:194] Registered new tserver with Master: 4da8afabe86e45a4a5cb8c7919aa721d (127.0.98.65:41921)
I20260926 20:47:32.368937 18984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.65:60995
W20260926 20:47:32.444303 19175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:32.444523 19175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:32.444557 19175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:32.445667 19175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:32.445725 19175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:32.445767 19175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:47:32.447072 19175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:32.447319 19175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:32.447595 19175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:32.452414 19183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:32.452513 19180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:32.452414 19181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:32.452682 19175 server_base.cc:1034] running on GCE node
I20260926 20:47:32.452908 19175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:32.453158 19175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:32.454317 19175 hybrid_clock.cc:648] HybridClock initialized: now 1790455652454308 us; error 30 us; skew 500 ppm
I20260926 20:47:32.455526 19175 webserver.cc:460] Webserver started at http://127.0.98.66:40771/ using document root <none> and password file <none>
I20260926 20:47:32.455828 19175 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:32.455893 19175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:32.456032 19175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:32.457077 19175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "5ce5cacb43ba4ddfa5605564084c5995"
format_stamp: "Formatted at 2026-09-26 20:47:32 on dist-test-slave-4hd3"
server_key: "eec6b9654b431a32c4b4a1e310d54e99"
server_key_iv: "63942d22e8fca5ffcac6095dd40afa76"
server_key_version: "encryptionkey@0"
I20260926 20:47:32.457437 19175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "5ce5cacb43ba4ddfa5605564084c5995"
format_stamp: "Formatted at 2026-09-26 20:47:32 on dist-test-slave-4hd3"
server_key: "eec6b9654b431a32c4b4a1e310d54e99"
server_key_iv: "63942d22e8fca5ffcac6095dd40afa76"
server_key_version: "encryptionkey@0"
I20260926 20:47:32.458767 19175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:47:32.459564 19189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:32.459774 19175 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:47:32.459878 19175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "5ce5cacb43ba4ddfa5605564084c5995"
format_stamp: "Formatted at 2026-09-26 20:47:32 on dist-test-slave-4hd3"
server_key: "eec6b9654b431a32c4b4a1e310d54e99"
server_key_iv: "63942d22e8fca5ffcac6095dd40afa76"
server_key_version: "encryptionkey@0"
I20260926 20:47:32.459964 19175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:32.479444 19175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:32.479827 19175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:32.479978 19175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:32.480216 19175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:32.480551 19175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:32.480602 19175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:32.480636 19175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:32.480662 19175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:32.486328 19175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:43197
I20260926 20:47:32.486375 19302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:43197 every 8 connection(s)
I20260926 20:47:32.486706 19175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:47:32.489250 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 19175
I20260926 20:47:32.489331 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:47:32.489558 393 external_mini_cluster.cc:1428] Setting key c4ec934f61693018ee9e8bc93aff64b3
I20260926 20:47:32.490667 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34117
--builtin_ntp_servers=127.0.98.84:43541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:47:32.493017 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.487503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:51223 (local address 127.0.98.126:34117)
0926 20:47:32.487569 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:32.487571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:32.487844 (+ 273us) server_negotiation.cc:407] Connection header received
0926 20:47:32.488420 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:32.488421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:32.488443 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:32.488523 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:32.489205 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.489918 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.491404 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.491592 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.491648 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.492723 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:32.492724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:32.492724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:32.492730 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:32.492747 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:32.492752 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:32.492814 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:32.492880 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 20:47:32.492910 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:47:32.493158 19305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.487497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34117 (local address 127.0.98.66:51223)
0926 20:47:32.487686 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:32.487755 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:32.488270 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:32.488599 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:32.488603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:32.488613 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:32.489107 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:32.489112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.489992 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.489993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:32.491321 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:32.491324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.491671 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.491673 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:32.491861 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.492576 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:47:32.492577 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:32.492610 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:47:32.492633 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:32.492820 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:32.492824 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:32.492827 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:47:32.492881 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:47:32.492960 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260926 20:47:32.493782 19303 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34117
I20260926 20:47:32.493881 19303 heartbeater.cc:464] Registering TS with master...
I20260926 20:47:32.494071 19303 heartbeater.cc:511] Master 127.0.98.126:34117 requested a full tablet report, sending...
I20260926 20:47:32.494426 18984 ts_manager.cc:194] Registered new tserver with Master: 5ce5cacb43ba4ddfa5605564084c5995 (127.0.98.66:43197)
I20260926 20:47:32.494801 18984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.66:51223
W20260926 20:47:32.572311 19306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:32.572539 19306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:32.572573 19306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:32.573721 19306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:32.573779 19306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:32.573819 19306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:47:32.575155 19306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:32.575409 19306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:32.575692 19306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:32.580792 19311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:32.580862 19306 server_base.cc:1034] running on GCE node
W20260926 20:47:32.580791 19314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:32.580982 19312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:32.581259 19306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:32.581467 19306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:32.582607 19306 hybrid_clock.cc:648] HybridClock initialized: now 1790455652582596 us; error 29 us; skew 500 ppm
I20260926 20:47:32.583964 19306 webserver.cc:460] Webserver started at http://127.0.98.67:43435/ using document root <none> and password file <none>
I20260926 20:47:32.584209 19306 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:32.584256 19306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:32.584415 19306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:32.585457 19306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "8e04af7b101b429f86507ab83cae4ea4"
format_stamp: "Formatted at 2026-09-26 20:47:32 on dist-test-slave-4hd3"
server_key: "cfd05b9a75dbcf261578b343481a45a6"
server_key_iv: "ab185033775a402adf6eba740855a691"
server_key_version: "encryptionkey@0"
I20260926 20:47:32.585858 19306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "8e04af7b101b429f86507ab83cae4ea4"
format_stamp: "Formatted at 2026-09-26 20:47:32 on dist-test-slave-4hd3"
server_key: "cfd05b9a75dbcf261578b343481a45a6"
server_key_iv: "ab185033775a402adf6eba740855a691"
server_key_version: "encryptionkey@0"
I20260926 20:47:32.587100 19306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:47:32.587819 19320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:32.587992 19306 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:47:32.588086 19306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "8e04af7b101b429f86507ab83cae4ea4"
format_stamp: "Formatted at 2026-09-26 20:47:32 on dist-test-slave-4hd3"
server_key: "cfd05b9a75dbcf261578b343481a45a6"
server_key_iv: "ab185033775a402adf6eba740855a691"
server_key_version: "encryptionkey@0"
I20260926 20:47:32.588168 19306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:32.607283 19306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:32.607642 19306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:32.607833 19306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:32.608098 19306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:32.608451 19306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:32.608510 19306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:32.608552 19306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:32.608595 19306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:32.614512 19306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:36685
I20260926 20:47:32.614554 19433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:36685 every 8 connection(s)
I20260926 20:47:32.614923 19306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:47:32.616185 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 19306
I20260926 20:47:32.616287 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:47:32.616482 393 external_mini_cluster.cc:1428] Setting key e5fa71b05ff1e50c3f52996962306f8c
I20260926 20:47:32.620589 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.615667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:48865 (local address 127.0.98.126:34117)
0926 20:47:32.615766 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:32.615768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:32.616000 (+ 232us) server_negotiation.cc:407] Connection header received
0926 20:47:32.616755 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:32.616756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:32.616770 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:32.616823 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:32.617654 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.618152 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.619158 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.619450 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.619508 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.620389 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:32.620390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:32.620390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:32.620394 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:32.620406 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:32.620410 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:32.620454 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:32.620503 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 20:47:32.620530 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 20:47:32.620826 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.615664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34117 (local address 127.0.98.67:48865)
0926 20:47:32.615873 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:32.615908 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:32.616606 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:32.616907 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:32.616912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:32.616925 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:32.617583 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:32.617588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.618226 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.618227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:32.619059 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:32.619061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.619526 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.619527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:32.619640 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.620262 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:47:32.620263 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:32.620292 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:47:32.620315 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:32.620454 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:32.620457 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:32.620459 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:47:32.620506 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 20:47:32.620581 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":117,"threads_started":1}
I20260926 20:47:32.621251 19434 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34117
I20260926 20:47:32.621344 19434 heartbeater.cc:464] Registering TS with master...
I20260926 20:47:32.621531 19434 heartbeater.cc:511] Master 127.0.98.126:34117 requested a full tablet report, sending...
I20260926 20:47:32.621902 18984 ts_manager.cc:194] Registered new tserver with Master: 8e04af7b101b429f86507ab83cae4ea4 (127.0.98.67:36685)
I20260926 20:47:32.622251 18984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.67:48865
I20260926 20:47:32.628397 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:47:32.632701 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.629727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55440 (local address 127.0.98.126:34117)
0926 20:47:32.629916 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:32.629918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:32.629927 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:47:32.629947 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:32.629947 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:32.629957 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:32.630003 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:32.630162 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.630657 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.631439 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.631625 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.631681 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.632379 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:32.632380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:32.632381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:32.632384 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:32.632398 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:32.632402 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:32.632505 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:32.632571 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 20:47:32.632582 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260926 20:47:32.634171 18984 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:47:32.635717 18984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:47:32.642025 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.637866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:36685 (local address 127.0.0.1:54808)
0926 20:47:32.638183 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:32.638194 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:32.638250 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:32.638634 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:32.638637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:32.638640 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:32.638707 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:32.638709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.639663 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.639664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:32.640904 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:32.640907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.641733 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.641734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:32.641812 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.641826 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 20:47:32.641899 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 20:47:32.641948 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":110,"threads_started":1}
I20260926 20:47:32.642249 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.637842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:43197 (local address 127.0.0.1:36974)
0926 20:47:32.638030 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:32.638067 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:32.638127 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:32.638721 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:32.638727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:32.638729 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:32.638790 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:32.638792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.640160 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.640161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:32.641149 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:32.641151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.642030 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.642031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:32.642103 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.642107 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:47:32.642143 (+ 36us) client_negotiation.cc:240] Negotiation successful
0926 20:47:32.642186 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":103,"threads_started":1}
I20260926 20:47:32.642052 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.638254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54808 (local address 127.0.98.67:36685)
0926 20:47:32.638466 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:32.638470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:32.638479 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:47:32.638513 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:32.638514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:32.638528 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:32.638786 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:32.638845 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.639569 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.640981 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.641640 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.641719 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.641808 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:32.641911 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 20:47:32.641953 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":121,"threads_started":1}
I20260926 20:47:32.642714 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.637988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36974 (local address 127.0.98.66:43197)
0926 20:47:32.638437 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:32.638443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:32.638453 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:47:32.638497 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:32.638499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:32.638517 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:32.638603 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:32.638879 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.640073 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.641222 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.641934 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.642435 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.642536 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:32.642560 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 20:47:32.642605 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":104,"threads_started":1}
I20260926 20:47:32.643035 19368 tablet_service.cc:1467] Processing CreateTablet for tablet b632c9e8c56c44c08acd225e02199e9e (DEFAULT_TABLE table=test-table [id=4027763eb1454898a9c5e731623cc2b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:47:32.643342 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.638107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41921 (local address 127.0.0.1:55460)
0926 20:47:32.638467 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:32.638482 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:32.638550 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:32.638812 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:32.638813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:32.638815 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:32.638849 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:32.638861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.641218 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.641219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:32.642039 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:32.642042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.643111 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.643112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:32.643174 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.643179 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:47:32.643237 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:47:32.643277 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":80,"threads_started":1}
I20260926 20:47:32.643437 19368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b632c9e8c56c44c08acd225e02199e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:32.643383 19237 tablet_service.cc:1467] Processing CreateTablet for tablet b632c9e8c56c44c08acd225e02199e9e (DEFAULT_TABLE table=test-table [id=4027763eb1454898a9c5e731623cc2b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:47:32.643616 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.638195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55460 (local address 127.0.98.65:41921)
0926 20:47:32.638395 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:32.638400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:32.638542 (+ 142us) server_negotiation.cc:407] Connection header received
0926 20:47:32.638662 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:32.638665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:32.638682 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:32.638751 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:32.638937 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.640072 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.642136 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.642907 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.642986 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.643073 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:32.643287 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 20:47:32.643336 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
I20260926 20:47:32.643729 19237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b632c9e8c56c44c08acd225e02199e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:32.646003 19448 tablet_bootstrap.cc:492] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4: Bootstrap starting.
I20260926 20:47:32.646142 19449 tablet_bootstrap.cc:492] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995: Bootstrap starting.
I20260926 20:47:32.646683 19448 tablet_bootstrap.cc:654] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:32.646878 19449 tablet_bootstrap.cc:654] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:32.646791 19106 tablet_service.cc:1467] Processing CreateTablet for tablet b632c9e8c56c44c08acd225e02199e9e (DEFAULT_TABLE table=test-table [id=4027763eb1454898a9c5e731623cc2b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:47:32.647017 19448 log.cc:824] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:32.647111 19106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b632c9e8c56c44c08acd225e02199e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:32.647219 19449 log.cc:824] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:32.647846 19448 tablet_bootstrap.cc:492] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4: No bootstrap required, opened a new log
I20260926 20:47:32.647926 19448 ts_tablet_manager.cc:1397] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:47:32.650586 19448 raft_consensus.cc:348] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5cacb43ba4ddfa5605564084c5995" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43197 } } peers { permanent_uuid: "8e04af7b101b429f86507ab83cae4ea4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36685 } } peers { permanent_uuid: "4da8afabe86e45a4a5cb8c7919aa721d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41921 } }
I20260926 20:47:32.650763 19448 raft_consensus.cc:374] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:32.650869 19448 raft_consensus.cc:729] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e04af7b101b429f86507ab83cae4ea4, State: Initialized, Role: FOLLOWER
I20260926 20:47:32.650971 19449 tablet_bootstrap.cc:492] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995: No bootstrap required, opened a new log
I20260926 20:47:32.651039 19449 ts_tablet_manager.cc:1397] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 20:47:32.651046 19448 consensus_queue.cc:260] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5cacb43ba4ddfa5605564084c5995" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43197 } } peers { permanent_uuid: "8e04af7b101b429f86507ab83cae4ea4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36685 } } peers { permanent_uuid: "4da8afabe86e45a4a5cb8c7919aa721d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41921 } }
I20260926 20:47:32.651317 19448 ts_tablet_manager.cc:1428] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:47:32.651471 19434 heartbeater.cc:503] Master 127.0.98.126:34117 was elected leader, sending a full tablet report...
I20260926 20:47:32.651816 19452 tablet_bootstrap.cc:492] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d: Bootstrap starting.
I20260926 20:47:32.652525 19452 tablet_bootstrap.cc:654] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:32.652845 19452 log.cc:824] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:32.653618 19452 tablet_bootstrap.cc:492] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d: No bootstrap required, opened a new log
I20260926 20:47:32.653704 19452 ts_tablet_manager.cc:1397] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:47:32.653960 19449 raft_consensus.cc:348] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5cacb43ba4ddfa5605564084c5995" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43197 } } peers { permanent_uuid: "8e04af7b101b429f86507ab83cae4ea4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36685 } } peers { permanent_uuid: "4da8afabe86e45a4a5cb8c7919aa721d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41921 } }
I20260926 20:47:32.654126 19449 raft_consensus.cc:374] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:32.654175 19449 raft_consensus.cc:729] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce5cacb43ba4ddfa5605564084c5995, State: Initialized, Role: FOLLOWER
I20260926 20:47:32.654311 19449 consensus_queue.cc:260] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5cacb43ba4ddfa5605564084c5995" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43197 } } peers { permanent_uuid: "8e04af7b101b429f86507ab83cae4ea4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36685 } } peers { permanent_uuid: "4da8afabe86e45a4a5cb8c7919aa721d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41921 } }
I20260926 20:47:32.654618 19449 ts_tablet_manager.cc:1428] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 20:47:32.654666 19303 heartbeater.cc:503] Master 127.0.98.126:34117 was elected leader, sending a full tablet report...
I20260926 20:47:32.656155 19452 raft_consensus.cc:348] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5cacb43ba4ddfa5605564084c5995" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43197 } } peers { permanent_uuid: "8e04af7b101b429f86507ab83cae4ea4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36685 } } peers { permanent_uuid: "4da8afabe86e45a4a5cb8c7919aa721d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41921 } }
I20260926 20:47:32.656297 19452 raft_consensus.cc:374] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:32.656343 19452 raft_consensus.cc:729] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da8afabe86e45a4a5cb8c7919aa721d, State: Initialized, Role: FOLLOWER
I20260926 20:47:32.656461 19452 consensus_queue.cc:260] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5cacb43ba4ddfa5605564084c5995" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43197 } } peers { permanent_uuid: "8e04af7b101b429f86507ab83cae4ea4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36685 } } peers { permanent_uuid: "4da8afabe86e45a4a5cb8c7919aa721d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41921 } }
I20260926 20:47:32.656698 19452 ts_tablet_manager.cc:1428] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 20:47:32.656770 19172 heartbeater.cc:503] Master 127.0.98.126:34117 was elected leader, sending a full tablet report...
I20260926 20:47:32.670845 19456 raft_consensus.cc:482] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:47:32.671017 19456 raft_consensus.cc:504] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5cacb43ba4ddfa5605564084c5995" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43197 } } peers { permanent_uuid: "8e04af7b101b429f86507ab83cae4ea4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36685 } } peers { permanent_uuid: "4da8afabe86e45a4a5cb8c7919aa721d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41921 } }
I20260926 20:47:32.671459 19456 leader_election.cc:288] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ce5cacb43ba4ddfa5605564084c5995 (127.0.98.66:43197), 8e04af7b101b429f86507ab83cae4ea4 (127.0.98.67:36685)
I20260926 20:47:32.674872 19174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.671648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:36685 (local address 127.0.98.65:54691)
0926 20:47:32.671704 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:32.671734 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:32.671780 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:32.671964 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:32.671965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:32.671967 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:32.672025 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:32.672026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.672858 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.672859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:32.674056 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:32.674059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.674687 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.674688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:32.674737 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.674740 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:47:32.674781 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 20:47:32.674814 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":4}
I20260926 20:47:32.674957 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.671748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:54691 (local address 127.0.98.67:36685)
0926 20:47:32.671816 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:32.671818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:32.671828 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:47:32.671860 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:32.671861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:32.671875 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:32.671944 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:32.672102 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.672763 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.674148 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.674608 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.674682 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.674692 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:32.674832 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 20:47:32.674880 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:47:32.675272 19388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b632c9e8c56c44c08acd225e02199e9e" candidate_uuid: "4da8afabe86e45a4a5cb8c7919aa721d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e04af7b101b429f86507ab83cae4ea4" is_pre_election: true
I20260926 20:47:32.675387 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.671874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:42023 (local address 127.0.98.66:43197)
0926 20:47:32.671953 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:32.671955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:32.672292 (+ 337us) server_negotiation.cc:407] Connection header received
0926 20:47:32.672329 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:32.672329 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:32.672339 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:32.672398 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:32.672578 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.673107 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.674385 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.674995 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.675065 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.675075 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:32.675285 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 20:47:32.675328 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:47:32.675395 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.672030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:43197 (local address 127.0.98.65:42023)
0926 20:47:32.672224 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:32.672233 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:32.672279 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:32.672445 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:32.672446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:32.672448 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:32.672510 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:32.672511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.673205 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.673206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:32.674314 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:32.674315 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.675125 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.675126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:32.675211 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.675214 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:47:32.675280 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 20:47:32.675316 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":8,"thread_start_us":124,"threads_started":1}
I20260926 20:47:32.675444 19388 raft_consensus.cc:2454] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da8afabe86e45a4a5cb8c7919aa721d in term 0.
I20260926 20:47:32.675669 19059 leader_election.cc:302] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da8afabe86e45a4a5cb8c7919aa721d, 8e04af7b101b429f86507ab83cae4ea4; no voters:
I20260926 20:47:32.675686 19257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b632c9e8c56c44c08acd225e02199e9e" candidate_uuid: "4da8afabe86e45a4a5cb8c7919aa721d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce5cacb43ba4ddfa5605564084c5995" is_pre_election: true
I20260926 20:47:32.675845 19257 raft_consensus.cc:2454] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da8afabe86e45a4a5cb8c7919aa721d in term 0.
I20260926 20:47:32.675856 19456 raft_consensus.cc:2781] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:47:32.675941 19456 raft_consensus.cc:482] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:47:32.675987 19456 raft_consensus.cc:3037] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:32.676692 19456 raft_consensus.cc:504] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5cacb43ba4ddfa5605564084c5995" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43197 } } peers { permanent_uuid: "8e04af7b101b429f86507ab83cae4ea4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36685 } } peers { permanent_uuid: "4da8afabe86e45a4a5cb8c7919aa721d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41921 } }
I20260926 20:47:32.676837 19456 leader_election.cc:288] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [CANDIDATE]: Term 1 election: Requested vote from peers 5ce5cacb43ba4ddfa5605564084c5995 (127.0.98.66:43197), 8e04af7b101b429f86507ab83cae4ea4 (127.0.98.67:36685)
I20260926 20:47:32.676981 19257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b632c9e8c56c44c08acd225e02199e9e" candidate_uuid: "4da8afabe86e45a4a5cb8c7919aa721d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce5cacb43ba4ddfa5605564084c5995"
I20260926 20:47:32.677044 19388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b632c9e8c56c44c08acd225e02199e9e" candidate_uuid: "4da8afabe86e45a4a5cb8c7919aa721d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e04af7b101b429f86507ab83cae4ea4"
I20260926 20:47:32.677080 19257 raft_consensus.cc:3037] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:32.677135 19388 raft_consensus.cc:3037] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:32.677831 19257 raft_consensus.cc:2454] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da8afabe86e45a4a5cb8c7919aa721d in term 1.
I20260926 20:47:32.677834 19388 raft_consensus.cc:2454] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da8afabe86e45a4a5cb8c7919aa721d in term 1.
I20260926 20:47:32.678078 19059 leader_election.cc:302] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da8afabe86e45a4a5cb8c7919aa721d, 8e04af7b101b429f86507ab83cae4ea4; no voters:
I20260926 20:47:32.678194 19456 raft_consensus.cc:2781] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:32.678368 19456 raft_consensus.cc:686] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [term 1 LEADER]: Becoming Leader. State: Replica: 4da8afabe86e45a4a5cb8c7919aa721d, State: Running, Role: LEADER
I20260926 20:47:32.678468 19456 consensus_queue.cc:237] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5cacb43ba4ddfa5605564084c5995" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43197 } } peers { permanent_uuid: "8e04af7b101b429f86507ab83cae4ea4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36685 } } peers { permanent_uuid: "4da8afabe86e45a4a5cb8c7919aa721d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41921 } }
I20260926 20:47:32.679216 18984 catalog_manager.cc:5495] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da8afabe86e45a4a5cb8c7919aa721d (127.0.98.65). New cstate: current_term: 1 leader_uuid: "4da8afabe86e45a4a5cb8c7919aa721d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5cacb43ba4ddfa5605564084c5995" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e04af7b101b429f86507ab83cae4ea4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da8afabe86e45a4a5cb8c7919aa721d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41921 } health_report { overall_health: HEALTHY } } }
I20260926 20:47:32.692703 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.690068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.0.98.65:41921)
0926 20:47:32.690133 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:32.690135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:32.690151 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:47:32.690183 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:32.690183 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:32.690201 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:32.690261 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:47:32.690381 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.690826 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.691808 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:32.692003 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:32.692067 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:32.692255 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:47:32.692347 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:47:32.692431 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:32.692571 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 20:47:32.692584 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:47:32.694583 19388 raft_consensus.cc:1261] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4 [term 1 FOLLOWER]: Refusing update from remote peer 4da8afabe86e45a4a5cb8c7919aa721d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:47:32.694587 19257 raft_consensus.cc:1261] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995 [term 1 FOLLOWER]: Refusing update from remote peer 4da8afabe86e45a4a5cb8c7919aa721d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:47:32.694954 19456 consensus_queue.cc:1035] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e04af7b101b429f86507ab83cae4ea4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:47:32.695101 19458 consensus_queue.cc:1035] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce5cacb43ba4ddfa5605564084c5995" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:47:32.698144 19463 mvcc.cc:205] Tried to move back new op lower bound from 7333706353434443776 to 7333706353371598848. Current Snapshot: MvccSnapshot[applied={T|T < 7333706353434443776 or (T in {7333706353434443776})}]
I20260926 20:47:32.700405 18982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:47:32.700503 18982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:47:32.701263 18982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7f58d82bbe143d0aea72319707a8974: Sending DeleteTablet for 3 replicas of tablet b632c9e8c56c44c08acd225e02199e9e
I20260926 20:47:32.701611 19237 tablet_service.cc:1514] Processing DeleteTablet for tablet b632c9e8c56c44c08acd225e02199e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:47:32 UTC) from {username='slave'} at 127.0.0.1:36974
I20260926 20:47:32.701854 19471 tablet_replica.cc:330] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995: stopping tablet replica
I20260926 20:47:32.701841 19368 tablet_service.cc:1514] Processing DeleteTablet for tablet b632c9e8c56c44c08acd225e02199e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:47:32 UTC) from {username='slave'} at 127.0.0.1:54808
I20260926 20:47:32.701954 19471 raft_consensus.cc:2229] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:47:32.702064 19471 raft_consensus.cc:2258] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:47:32.702149 19106 tablet_service.cc:1514] Processing DeleteTablet for tablet b632c9e8c56c44c08acd225e02199e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:47:32 UTC) from {username='slave'} at 127.0.0.1:55460
I20260926 20:47:32.702353 19473 tablet_replica.cc:330] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d: stopping tablet replica
I20260926 20:47:32.702484 19473 raft_consensus.cc:2229] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:47:32.702531 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 19044
I20260926 20:47:32.702525 19471 ts_tablet_manager.cc:1905] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:47:32.702692 19473 raft_consensus.cc:2258] T b632c9e8c56c44c08acd225e02199e9e P 4da8afabe86e45a4a5cb8c7919aa721d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:47:32.704538 19471 ts_tablet_manager.cc:1918] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:47:32.704612 19471 log.cc:1196] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/b632c9e8c56c44c08acd225e02199e9e
I20260926 20:47:32.704949 19471 ts_tablet_manager.cc:1939] T b632c9e8c56c44c08acd225e02199e9e P 5ce5cacb43ba4ddfa5605564084c5995: Deleting consensus metadata
I20260926 20:47:32.705392 18969 catalog_manager.cc:4841] TS 5ce5cacb43ba4ddfa5605564084c5995 (127.0.98.66:43197): tablet b632c9e8c56c44c08acd225e02199e9e (table test-table [id=4027763eb1454898a9c5e731623cc2b8]) successfully deleted
I20260926 20:47:32.707863 19472 tablet_replica.cc:330] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4: stopping tablet replica
I20260926 20:47:32.708000 19472 raft_consensus.cc:2229] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:47:32.708122 19472 raft_consensus.cc:2258] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:47:32.708527 19472 ts_tablet_manager.cc:1905] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:47:32.709787 18969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:41921 (error 108)
I20260926 20:47:32.710075 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 19175
I20260926 20:47:32.710160 19472 ts_tablet_manager.cc:1918] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:47:32.710228 19472 log.cc:1196] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/b632c9e8c56c44c08acd225e02199e9e
I20260926 20:47:32.710448 19472 ts_tablet_manager.cc:1939] T b632c9e8c56c44c08acd225e02199e9e P 8e04af7b101b429f86507ab83cae4ea4: Deleting consensus metadata
I20260926 20:47:32.710649 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:32.710157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41921 (local address 127.0.0.1:55484)
0926 20:47:32.710563 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:32.710607 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383}
W20260926 20:47:32.710888 18969 catalog_manager.cc:4584] TS 4da8afabe86e45a4a5cb8c7919aa721d (127.0.98.65:41921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b632c9e8c56c44c08acd225e02199e9e: Network error: Client connection negotiation failed: client connection to 127.0.98.65:41921: connect: Connection refused (error 111)
I20260926 20:47:32.712286 18968 catalog_manager.cc:4841] TS 8e04af7b101b429f86507ab83cae4ea4 (127.0.98.67:36685): tablet b632c9e8c56c44c08acd225e02199e9e (table test-table [id=4027763eb1454898a9c5e731623cc2b8]) successfully deleted
I20260926 20:47:32.715271 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 19306
I20260926 20:47:32.719769 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 18953
2026-09-26T20:47:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (692 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:32 dist-test-slave-4hd3 krb5kdc[19477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:32 dist-test-slave-4hd3 krb5kdc[19477](info): set up 2 sockets
Sep 26 20:47:32 dist-test-slave-4hd3 krb5kdc[19477](info): commencing operation
krb5kdc: starting...
W20260926 20:47:33.759469 393 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:33 dist-test-slave-4hd3 krb5kdc[19477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:47:33.872566 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:44061
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:44061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:33.954293 19490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:33.954734 19490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:33.954792 19490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:33.956115 19490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:33.956162 19490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:33.956176 19490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:33.956189 19490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:33.956202 19490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:33.957662 19490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:44061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:44061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:33.957873 19490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:33.958060 19490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:33.963188 19495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:33.963176 19496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:33.963196 19490 server_base.cc:1034] running on GCE node
W20260926 20:47:33.963164 19498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:33.963619 19490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:33.963973 19490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:33.965149 19490 hybrid_clock.cc:648] HybridClock initialized: now 1790455653965127 us; error 63 us; skew 500 ppm
Sep 26 20:47:33 dist-test-slave-4hd3 krb5kdc[19477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:33.967783 19490 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:47:33.968281 19490 webserver.cc:460] Webserver started at http://127.0.98.126:46115/ using document root <none> and password file <none>
I20260926 20:47:33.968530 19490 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:33.968601 19490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:33.968753 19490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:33.969841 19490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "2084f5310e5340d0a30e0daf463cb55e"
format_stamp: "Formatted at 2026-09-26 20:47:33 on dist-test-slave-4hd3"
server_key: "b98ce73f07a927b54b155f69ca755440"
server_key_iv: "1f3656f938de1f878d03494b07dcd6a3"
server_key_version: "encryptionkey@0"
I20260926 20:47:33.970213 19490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "2084f5310e5340d0a30e0daf463cb55e"
format_stamp: "Formatted at 2026-09-26 20:47:33 on dist-test-slave-4hd3"
server_key: "b98ce73f07a927b54b155f69ca755440"
server_key_iv: "1f3656f938de1f878d03494b07dcd6a3"
server_key_version: "encryptionkey@0"
I20260926 20:47:33.971577 19490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:47:33.972468 19505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:33.972700 19490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:33.972818 19490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "2084f5310e5340d0a30e0daf463cb55e"
format_stamp: "Formatted at 2026-09-26 20:47:33 on dist-test-slave-4hd3"
server_key: "b98ce73f07a927b54b155f69ca755440"
server_key_iv: "1f3656f938de1f878d03494b07dcd6a3"
server_key_version: "encryptionkey@0"
I20260926 20:47:33.972918 19490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:33.986212 19490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:33.988160 19490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:33.988322 19490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:33.992415 19490 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:44061
I20260926 20:47:33.992449 19557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:44061 every 8 connection(s)
I20260926 20:47:33.992787 19490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:33.993619 19558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:33.995954 19558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e: Bootstrap starting.
I20260926 20:47:33.996521 19558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:33.996799 19558 log.cc:824] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:33.997105 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 19490
I20260926 20:47:33.997228 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:33.997408 393 external_mini_cluster.cc:1428] Setting key 93a6cd152d830d9f613f7543e05f7e6a
I20260926 20:47:33.997619 19558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e: No bootstrap required, opened a new log
I20260926 20:47:33.999814 19558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2084f5310e5340d0a30e0daf463cb55e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44061 } }
I20260926 20:47:34.000025 19558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:34.000072 19558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2084f5310e5340d0a30e0daf463cb55e, State: Initialized, Role: FOLLOWER
I20260926 20:47:34.000204 19558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2084f5310e5340d0a30e0daf463cb55e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44061 } }
I20260926 20:47:34.000295 19558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:34.000340 19558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:34.000388 19558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:34.001119 19558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2084f5310e5340d0a30e0daf463cb55e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44061 } }
I20260926 20:47:34.001230 19558 leader_election.cc:302] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2084f5310e5340d0a30e0daf463cb55e; no voters:
I20260926 20:47:34.001376 19558 leader_election.cc:288] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:34.001432 19563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:34.001601 19558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:34.001622 19563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e [term 1 LEADER]: Becoming Leader. State: Replica: 2084f5310e5340d0a30e0daf463cb55e, State: Running, Role: LEADER
I20260926 20:47:34.001754 19563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2084f5310e5340d0a30e0daf463cb55e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44061 } }
I20260926 20:47:34.002154 19564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2084f5310e5340d0a30e0daf463cb55e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2084f5310e5340d0a30e0daf463cb55e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44061 } } }
I20260926 20:47:34.002381 19564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:34.002326 19565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2084f5310e5340d0a30e0daf463cb55e. Latest consensus state: current_term: 1 leader_uuid: "2084f5310e5340d0a30e0daf463cb55e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2084f5310e5340d0a30e0daf463cb55e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44061 } } }
I20260926 20:47:34.002626 19565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:34.002781 19567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:34.003268 19567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:34.005458 19567 catalog_manager.cc:1347] Generated new cluster ID: 1ac3e985776d4ae9bd36daebd07918d1
I20260926 20:47:34.005559 19567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:47:34 dist-test-slave-4hd3 krb5kdc[19477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:47:34.010353 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:33.997909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42936 (local address 127.0.98.126:44061)
0926 20:47:33.998108 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:33.998118 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:33.998152 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:47:33.998983 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:33.998993 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:33.999057 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:33.999213 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:33.999906 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.001674 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.002789 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.003412 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.003524 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:34.006516 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:34.006520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:34.006520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:34.006532 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:34.008658 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:34.009062 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:34.009065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:34.009065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:34.009176 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:34.009381 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:34.009382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:34.009382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:34.009758 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:34.009924 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:34.010006 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 20:47:34.010147 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260926 20:47:34.019467 19567 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:34.020080 19567 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:34.035912 19567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e: Generated new TSK 0
I20260926 20:47:34.036131 19567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260926 20:47:34.073539 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:44061
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:34.153549 19584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:34.153756 19584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:34.153780 19584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:34.154940 19584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:34.154983 19584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:34.155009 19584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:47:34.156571 19584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:44061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:34.156813 19584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:34.157059 19584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:34.162003 19592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:34.162003 19589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:34.162138 19590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:34.162392 19584 server_base.cc:1034] running on GCE node
I20260926 20:47:34.162607 19584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:34.162828 19584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:34.163991 19584 hybrid_clock.cc:648] HybridClock initialized: now 1790455654163983 us; error 38 us; skew 500 ppm
Sep 26 20:47:34 dist-test-slave-4hd3 krb5kdc[19477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:34.166513 19584 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260926 20:47:34.166963 19584 webserver.cc:460] Webserver started at http://127.0.98.65:34925/ using document root <none> and password file <none>
I20260926 20:47:34.167160 19584 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:34.167200 19584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:34.167287 19584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:34.168332 19584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "102b3ca0317848c4afaa2c46825a60c4"
format_stamp: "Formatted at 2026-09-26 20:47:34 on dist-test-slave-4hd3"
server_key: "babfb9f2fc6b2ba04472ca0e2beb61b6"
server_key_iv: "cee153c50a73ac3f3af6b122b0e90c05"
server_key_version: "encryptionkey@0"
I20260926 20:47:34.168634 19584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "102b3ca0317848c4afaa2c46825a60c4"
format_stamp: "Formatted at 2026-09-26 20:47:34 on dist-test-slave-4hd3"
server_key: "babfb9f2fc6b2ba04472ca0e2beb61b6"
server_key_iv: "cee153c50a73ac3f3af6b122b0e90c05"
server_key_version: "encryptionkey@0"
I20260926 20:47:34.169857 19584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:47:34.170507 19599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:34.170680 19584 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:47:34.170746 19584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "102b3ca0317848c4afaa2c46825a60c4"
format_stamp: "Formatted at 2026-09-26 20:47:34 on dist-test-slave-4hd3"
server_key: "babfb9f2fc6b2ba04472ca0e2beb61b6"
server_key_iv: "cee153c50a73ac3f3af6b122b0e90c05"
server_key_version: "encryptionkey@0"
I20260926 20:47:34.170797 19584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:34.195537 19584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:34.197845 19584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:34.197959 19584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:34.198133 19584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:34.198480 19584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:34.198523 19584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:34.198545 19584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:34.198558 19584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:34.204581 19584 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:38107
I20260926 20:47:34.204629 19712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:38107 every 8 connection(s)
I20260926 20:47:34.204926 19584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:47:34.208870 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 19584
I20260926 20:47:34.208978 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:47:34.209167 393 external_mini_cluster.cc:1428] Setting key 909593d8d641018a6e58e02401c14b9c
Sep 26 20:47:34 dist-test-slave-4hd3 krb5kdc[19477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:47:34.214847 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:34.205573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:44061 (local address 127.0.98.65:57229)
0926 20:47:34.205741 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:34.205777 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:34.206692 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:34.206941 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:34.206946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:34.207202 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:34.207384 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:34.207388 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.208151 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.208152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:34.208963 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:34.208965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.209452 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.209453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:34.209591 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:34.210089 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:47:34.210097 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:34.211601 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:34.213547 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:34.213553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:34.213554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:34.213759 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:34.213974 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:34.213974 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:34.213974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:34.214074 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:34.214375 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:34.214379 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:34.214494 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 20:47:34.214578 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:47:34.214695 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260926 20:47:34.214887 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:34.205594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:57229 (local address 127.0.98.126:44061)
0926 20:47:34.205642 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:34.205644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:34.205861 (+ 217us) server_negotiation.cc:407] Connection header received
0926 20:47:34.206809 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:34.206810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:34.206823 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:34.206871 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:34.207497 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.208061 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.209114 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.209354 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.209442 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:34.211694 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:34.211696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:34.211697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:34.211706 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:34.213454 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:34.213833 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:34.213834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:34.213835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:34.213896 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:34.214141 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:34.214142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:34.214142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:34.214303 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:34.214382 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:34.214651 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 20:47:34.214752 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:47:34.215639 19713 heartbeater.cc:347] Connected to a master server at 127.0.98.126:44061
I20260926 20:47:34.215785 19713 heartbeater.cc:464] Registering TS with master...
I20260926 20:47:34.215958 19713 heartbeater.cc:511] Master 127.0.98.126:44061 requested a full tablet report, sending...
I20260926 20:47:34.216482 19522 ts_manager.cc:194] Registered new tserver with Master: 102b3ca0317848c4afaa2c46825a60c4 (127.0.98.65:38107)
I20260926 20:47:34.217089 19522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:57229
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260926 20:47:34.231930 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:44061
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:34.311461 19718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:34.311695 19718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:34.311748 19718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:34.312888 19718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:34.312955 19718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:34.313000 19718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:47:34.314466 19718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:44061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:34.314718 19718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:34.314934 19718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:34.319908 19723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:34.320020 19718 server_base.cc:1034] running on GCE node
W20260926 20:47:34.320022 19726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:34.319917 19724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:34.320390 19718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:34.320608 19718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:34.321743 19718 hybrid_clock.cc:648] HybridClock initialized: now 1790455654321731 us; error 27 us; skew 500 ppm
Sep 26 20:47:34 dist-test-slave-4hd3 krb5kdc[19477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:34.324268 19718 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260926 20:47:34.324723 19718 webserver.cc:460] Webserver started at http://127.0.98.66:34801/ using document root <none> and password file <none>
I20260926 20:47:34.324895 19718 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:34.324936 19718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:34.325019 19718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:34.325968 19718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "78e03f8c8b5541abbd799ce17578ccf1"
format_stamp: "Formatted at 2026-09-26 20:47:34 on dist-test-slave-4hd3"
server_key: "0a0347dc859bbe6799540b59ebbf121a"
server_key_iv: "b908ef50fee658ff5a26d218d449c436"
server_key_version: "encryptionkey@0"
I20260926 20:47:34.326260 19718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "78e03f8c8b5541abbd799ce17578ccf1"
format_stamp: "Formatted at 2026-09-26 20:47:34 on dist-test-slave-4hd3"
server_key: "0a0347dc859bbe6799540b59ebbf121a"
server_key_iv: "b908ef50fee658ff5a26d218d449c436"
server_key_version: "encryptionkey@0"
I20260926 20:47:34.327539 19718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:47:34.328325 19733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:34.328518 19718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:47:34.328615 19718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "78e03f8c8b5541abbd799ce17578ccf1"
format_stamp: "Formatted at 2026-09-26 20:47:34 on dist-test-slave-4hd3"
server_key: "0a0347dc859bbe6799540b59ebbf121a"
server_key_iv: "b908ef50fee658ff5a26d218d449c436"
server_key_version: "encryptionkey@0"
I20260926 20:47:34.328692 19718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:34.338270 19718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:34.340200 19718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:34.340360 19718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:34.340590 19718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:34.340886 19718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:34.340938 19718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:34.340978 19718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:34.341008 19718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:34.346724 19718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:40639
I20260926 20:47:34.346767 19846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:40639 every 8 connection(s)
I20260926 20:47:34.347105 19718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:47:34 dist-test-slave-4hd3 krb5kdc[19477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:47:34.356251 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:34.347872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:56169 (local address 127.0.98.126:44061)
0926 20:47:34.347953 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:34.347956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:34.348176 (+ 220us) server_negotiation.cc:407] Connection header received
0926 20:47:34.349127 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:34.349129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:34.349142 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:34.349196 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:34.349849 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.350362 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.351353 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.351561 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.351650 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:34.353697 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:34.353699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:34.353699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:34.353707 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:34.354869 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:34.355276 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:34.355277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:34.355278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:34.355335 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:34.355578 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:34.355578 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:34.355579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:34.355693 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:34.355808 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:34.356078 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 20:47:34.356158 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:47:34.356544 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 19718
I20260926 20:47:34.356653 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:47:34.356824 393 external_mini_cluster.cc:1428] Setting key 20296df6afb1944db37e2173c1953830
I20260926 20:47:34.357060 19849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:34.347880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:44061 (local address 127.0.98.66:56169)
0926 20:47:34.348055 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:34.348090 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:34.349018 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:34.349265 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:34.349268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:34.349563 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:34.349751 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:34.349756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.350470 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.350471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:34.351264 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:34.351266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.351661 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.351662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:34.351804 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:34.352259 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:47:34.352267 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:34.353601 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:34.354973 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:34.354979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:34.354981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:34.355189 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:34.355401 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:34.355402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:34.355403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:34.355509 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:34.355801 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:34.355808 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:34.355948 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 20:47:34.356056 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 20:47:34.356208 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260926 20:47:34.357724 19847 heartbeater.cc:347] Connected to a master server at 127.0.98.126:44061
I20260926 20:47:34.357822 19847 heartbeater.cc:464] Registering TS with master...
I20260926 20:47:34.358031 19847 heartbeater.cc:511] Master 127.0.98.126:44061 requested a full tablet report, sending...
I20260926 20:47:34.358434 19522 ts_manager.cc:194] Registered new tserver with Master: 78e03f8c8b5541abbd799ce17578ccf1 (127.0.98.66:40639)
I20260926 20:47:34.358958 19522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:56169
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260926 20:47:34.380570 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:44061
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:34.461130 19852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:34.461337 19852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:34.461361 19852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:34.462461 19852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:34.462504 19852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:34.462529 19852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:47:34.464001 19852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:44061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:34.464177 19852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:34.464346 19852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:34.469421 19858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:34.469563 19852 server_base.cc:1034] running on GCE node
W20260926 20:47:34.469453 19860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:34.469508 19857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:34.469938 19852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:34.470216 19852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:34.471407 19852 hybrid_clock.cc:648] HybridClock initialized: now 1790455654471397 us; error 38 us; skew 500 ppm
Sep 26 20:47:34 dist-test-slave-4hd3 krb5kdc[19477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:34.474197 19852 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260926 20:47:34.474697 19852 webserver.cc:460] Webserver started at http://127.0.98.67:44793/ using document root <none> and password file <none>
I20260926 20:47:34.474946 19852 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:34.475015 19852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:34.475160 19852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:34.476302 19852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "00a742844d0f42fea77730ffe165f631"
format_stamp: "Formatted at 2026-09-26 20:47:34 on dist-test-slave-4hd3"
server_key: "c6ca71d4f4d9431511282156ca20e3b8"
server_key_iv: "51c743ef77b574da580b1fe73cfcb7c0"
server_key_version: "encryptionkey@0"
I20260926 20:47:34.476719 19852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "00a742844d0f42fea77730ffe165f631"
format_stamp: "Formatted at 2026-09-26 20:47:34 on dist-test-slave-4hd3"
server_key: "c6ca71d4f4d9431511282156ca20e3b8"
server_key_iv: "51c743ef77b574da580b1fe73cfcb7c0"
server_key_version: "encryptionkey@0"
I20260926 20:47:34.478021 19852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:47:34.478765 19867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:34.479008 19852 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:34.479125 19852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "00a742844d0f42fea77730ffe165f631"
format_stamp: "Formatted at 2026-09-26 20:47:34 on dist-test-slave-4hd3"
server_key: "c6ca71d4f4d9431511282156ca20e3b8"
server_key_iv: "51c743ef77b574da580b1fe73cfcb7c0"
server_key_version: "encryptionkey@0"
I20260926 20:47:34.479200 19852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:34.496670 19852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:34.499161 19852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:34.499337 19852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:34.499595 19852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:34.499981 19852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:34.500048 19852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:34.500093 19852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:34.500119 19852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:34.506388 19852 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:38599
I20260926 20:47:34.506430 19980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:38599 every 8 connection(s)
I20260926 20:47:34.506768 19852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:47:34 dist-test-slave-4hd3 krb5kdc[19477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:47:34.515398 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 19852
I20260926 20:47:34.515506 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:47:34.515746 393 external_mini_cluster.cc:1428] Setting key ece05bfedef3693f3b020b7ce00ac992
I20260926 20:47:34.515937 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:34.507489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:44177 (local address 127.0.98.126:44061)
0926 20:47:34.507547 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:34.507549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:34.507784 (+ 235us) server_negotiation.cc:407] Connection header received
0926 20:47:34.508739 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:34.508741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:34.508752 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:34.508807 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:34.509420 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.509922 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.510894 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.511078 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.511133 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:34.513250 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:34.513251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:34.513251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:34.513258 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:34.514633 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:34.514966 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:34.514967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:34.514967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:34.515016 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:34.515218 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:34.515218 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:34.515218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:34.515312 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:34.515391 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:34.515753 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 20:47:34.515832 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 20:47:34.516170 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:34.507481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:44061 (local address 127.0.98.67:44177)
0926 20:47:34.507627 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:34.507663 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:34.508640 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:34.508870 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:34.508874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:34.509151 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:34.509342 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:34.509347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.509998 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.509999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:34.510807 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:34.510809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.511149 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.511149 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:34.511246 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:34.511808 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:47:34.511817 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:34.513167 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:34.514702 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:34.514706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:34.514708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:34.514904 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:34.515074 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:47:34.515074 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:47:34.515075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:47:34.515165 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:47:34.515405 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:34.515411 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:34.515581 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 20:47:34.515710 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 20:47:34.515896 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260926 20:47:34.516757 19981 heartbeater.cc:347] Connected to a master server at 127.0.98.126:44061
I20260926 20:47:34.516847 19981 heartbeater.cc:464] Registering TS with master...
I20260926 20:47:34.516986 19981 heartbeater.cc:511] Master 127.0.98.126:44061 requested a full tablet report, sending...
I20260926 20:47:34.517313 19522 ts_manager.cc:194] Registered new tserver with Master: 00a742844d0f42fea77730ffe165f631 (127.0.98.67:38599)
I20260926 20:47:34.517637 19522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:44177
I20260926 20:47:34.527478 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:47:34.534698 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:34.528831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42946 (local address 127.0.98.126:44061)
0926 20:47:34.528904 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:34.528906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:34.528939 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:47:34.528999 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:34.528999 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:34.529007 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:34.529058 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:34.529548 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.530018 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.530735 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.530928 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.530991 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:34.532289 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:34.532290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:34.532291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:34.532300 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:34.533600 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:34.533933 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:34.533934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:34.533934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:34.533984 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:34.534155 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:34.534156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:34.534156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:34.534290 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:34.534362 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:34.534523 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 20:47:34.534583 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:47:34.536592 19522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:47:34.538141 19522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:47:34.544569 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:34.540337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:40639 (local address 127.0.0.1:46182)
0926 20:47:34.540569 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:34.540612 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:34.540676 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:34.540941 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:34.540944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:34.540946 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:34.541016 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:34.541018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.542281 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.542282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:34.543398 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:34.543401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.544254 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.544255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:34.544339 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:34.544355 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 20:47:34.544433 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:47:34.544492 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":61,"thread_start_us":72,"threads_started":1}
I20260926 20:47:34.544649 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:34.540730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50220 (local address 127.0.98.67:38599)
0926 20:47:34.541051 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:34.541056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:34.541066 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:47:34.541101 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:34.541103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:34.541119 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:34.541191 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:34.541468 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.542120 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.543764 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.544288 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.544357 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:34.544412 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:34.544514 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 20:47:34.544556 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":81,"threads_started":1}
I20260926 20:47:34.544956 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:34.540849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34806 (local address 127.0.98.65:38107)
0926 20:47:34.541039 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:34.541044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:34.541054 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:47:34.541112 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:34.541115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:34.541138 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:34.541213 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:34.541437 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.542492 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.543778 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.544481 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.544540 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:34.544591 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:34.544867 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 20:47:34.544903 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":74,"threads_started":1}
I20260926 20:47:34.545138 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:34.540463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.0.98.66:40639)
0926 20:47:34.540642 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:34.540647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:34.540659 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:47:34.540804 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:34.540807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:34.540850 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:34.540936 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:34.541089 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.542164 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.543478 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.544163 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.544238 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:34.544294 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:34.544987 (+ 693us) server_negotiation.cc:299] Negotiation successful
0926 20:47:34.545061 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":111,"threads_started":1}
I20260926 20:47:34.544667 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:34.540405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:38599 (local address 127.0.0.1:50220)
0926 20:47:34.540742 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:34.540753 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:34.540807 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:34.541309 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:34.541309 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:34.541311 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:34.541360 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:34.541362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.542223 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.542224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:34.543647 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:34.543650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.544362 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.544363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:34.544431 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:34.544435 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:47:34.544501 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 20:47:34.544551 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":80,"thread_start_us":109,"threads_started":1}
I20260926 20:47:34.546077 19781 tablet_service.cc:1467] Processing CreateTablet for tablet debb3d070d9241e19adaf1ab24eef42f (DEFAULT_TABLE table=test-table [id=ca2af901be974eb9998cc23f287c469f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:47:34.546077 19915 tablet_service.cc:1467] Processing CreateTablet for tablet debb3d070d9241e19adaf1ab24eef42f (DEFAULT_TABLE table=test-table [id=ca2af901be974eb9998cc23f287c469f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:47:34.546444 19781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet debb3d070d9241e19adaf1ab24eef42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:34.546444 19915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet debb3d070d9241e19adaf1ab24eef42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:34.544809 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:34.540621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:38107 (local address 127.0.0.1:34806)
0926 20:47:34.540890 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:34.540899 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:34.540941 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:34.541292 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:34.541293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:34.541295 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:34.541357 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:34.541359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.542594 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.542595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:34.543686 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:34.543688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.544624 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.544625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:34.544679 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:34.544683 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:47:34.544732 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 20:47:34.544765 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":4,"thread_start_us":73,"threads_started":1}
I20260926 20:47:34.549835 19999 tablet_bootstrap.cc:492] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631: Bootstrap starting.
I20260926 20:47:34.549835 19998 tablet_bootstrap.cc:492] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1: Bootstrap starting.
I20260926 20:47:34.550549 19998 tablet_bootstrap.cc:654] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:34.550549 19999 tablet_bootstrap.cc:654] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:34.550498 19647 tablet_service.cc:1467] Processing CreateTablet for tablet debb3d070d9241e19adaf1ab24eef42f (DEFAULT_TABLE table=test-table [id=ca2af901be974eb9998cc23f287c469f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:47:34.550812 19647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet debb3d070d9241e19adaf1ab24eef42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:34.551174 19999 log.cc:824] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:34.551174 19998 log.cc:824] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:34.552209 19998 tablet_bootstrap.cc:492] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1: No bootstrap required, opened a new log
I20260926 20:47:34.552203 19999 tablet_bootstrap.cc:492] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631: No bootstrap required, opened a new log
I20260926 20:47:34.552294 19999 ts_tablet_manager.cc:1397] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:47:34.552297 19998 ts_tablet_manager.cc:1397] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:47:34.553658 19998 raft_consensus.cc:348] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e03f8c8b5541abbd799ce17578ccf1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 40639 } } peers { permanent_uuid: "00a742844d0f42fea77730ffe165f631" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38599 } } peers { permanent_uuid: "102b3ca0317848c4afaa2c46825a60c4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38107 } }
I20260926 20:47:34.553819 19998 raft_consensus.cc:374] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:34.553894 19998 raft_consensus.cc:729] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e03f8c8b5541abbd799ce17578ccf1, State: Initialized, Role: FOLLOWER
I20260926 20:47:34.554025 19998 consensus_queue.cc:260] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e03f8c8b5541abbd799ce17578ccf1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 40639 } } peers { permanent_uuid: "00a742844d0f42fea77730ffe165f631" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38599 } } peers { permanent_uuid: "102b3ca0317848c4afaa2c46825a60c4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38107 } }
I20260926 20:47:34.554330 19847 heartbeater.cc:503] Master 127.0.98.126:44061 was elected leader, sending a full tablet report...
I20260926 20:47:34.554334 19998 ts_tablet_manager.cc:1428] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 20:47:34.554351 20002 tablet_bootstrap.cc:492] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4: Bootstrap starting.
I20260926 20:47:34.554381 19999 raft_consensus.cc:348] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e03f8c8b5541abbd799ce17578ccf1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 40639 } } peers { permanent_uuid: "00a742844d0f42fea77730ffe165f631" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38599 } } peers { permanent_uuid: "102b3ca0317848c4afaa2c46825a60c4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38107 } }
I20260926 20:47:34.554524 19999 raft_consensus.cc:374] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:34.554554 19999 raft_consensus.cc:729] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a742844d0f42fea77730ffe165f631, State: Initialized, Role: FOLLOWER
I20260926 20:47:34.554667 19999 consensus_queue.cc:260] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e03f8c8b5541abbd799ce17578ccf1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 40639 } } peers { permanent_uuid: "00a742844d0f42fea77730ffe165f631" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38599 } } peers { permanent_uuid: "102b3ca0317848c4afaa2c46825a60c4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38107 } }
I20260926 20:47:34.554924 19999 ts_tablet_manager.cc:1428] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:47:34.554939 19981 heartbeater.cc:503] Master 127.0.98.126:44061 was elected leader, sending a full tablet report...
I20260926 20:47:34.555269 20002 tablet_bootstrap.cc:654] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:34.555920 20002 log.cc:824] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:34.556602 20002 tablet_bootstrap.cc:492] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4: No bootstrap required, opened a new log
I20260926 20:47:34.556654 20002 ts_tablet_manager.cc:1397] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:47:34.557976 20002 raft_consensus.cc:348] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e03f8c8b5541abbd799ce17578ccf1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 40639 } } peers { permanent_uuid: "00a742844d0f42fea77730ffe165f631" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38599 } } peers { permanent_uuid: "102b3ca0317848c4afaa2c46825a60c4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38107 } }
I20260926 20:47:34.558106 20002 raft_consensus.cc:374] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:34.558127 20002 raft_consensus.cc:729] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 102b3ca0317848c4afaa2c46825a60c4, State: Initialized, Role: FOLLOWER
I20260926 20:47:34.558238 20002 consensus_queue.cc:260] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e03f8c8b5541abbd799ce17578ccf1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 40639 } } peers { permanent_uuid: "00a742844d0f42fea77730ffe165f631" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38599 } } peers { permanent_uuid: "102b3ca0317848c4afaa2c46825a60c4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38107 } }
I20260926 20:47:34.558521 20002 ts_tablet_manager.cc:1428] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:47:34.558553 19713 heartbeater.cc:503] Master 127.0.98.126:44061 was elected leader, sending a full tablet report...
W20260926 20:47:34.597695 19848 tablet.cc:2369] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:47:34.705659 19714 tablet.cc:2369] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:47:34.757377 19982 tablet.cc:2369] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:47:34.865444 20006 raft_consensus.cc:482] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:47:34.865617 20006 raft_consensus.cc:504] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e03f8c8b5541abbd799ce17578ccf1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 40639 } } peers { permanent_uuid: "00a742844d0f42fea77730ffe165f631" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38599 } } peers { permanent_uuid: "102b3ca0317848c4afaa2c46825a60c4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38107 } }
I20260926 20:47:34.866045 20006 leader_election.cc:288] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78e03f8c8b5541abbd799ce17578ccf1 (127.0.98.66:40639), 00a742844d0f42fea77730ffe165f631 (127.0.98.67:38599)
I20260926 20:47:34.869261 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:34.866294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:57969 (local address 127.0.98.66:40639)
0926 20:47:34.866343 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:34.866346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:34.866493 (+ 147us) server_negotiation.cc:407] Connection header received
0926 20:47:34.866536 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:34.866537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:34.866550 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:34.866607 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:34.866790 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.867301 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.868508 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.868879 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.868934 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:34.868942 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:34.869144 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 20:47:34.869186 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 20:47:34.869269 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:34.866352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:41691 (local address 127.0.98.67:38599)
0926 20:47:34.866466 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:34.866468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:34.866500 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:47:34.866579 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:34.866600 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:34.866610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:34.866665 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:34.866895 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.867401 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.868501 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.868881 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.869003 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:34.869014 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:34.869162 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 20:47:34.869202 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 20:47:34.869283 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:34.866256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:40639 (local address 127.0.98.65:57969)
0926 20:47:34.866398 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:34.866409 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:34.866483 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:34.866629 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:34.866630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:34.866632 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:34.866717 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:34.866719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.867405 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.867407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:34.868379 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:34.868381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.868987 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.868988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:34.869065 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:34.869068 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:47:34.869137 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:47:34.869183 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":45,"thread_start_us":132,"threads_started":1}
I20260926 20:47:34.869289 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:34.866270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:38599 (local address 127.0.98.65:41691)
0926 20:47:34.866430 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:34.866442 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:34.866518 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:34.866681 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:34.866682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:34.866684 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:34.866785 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:34.866787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.867489 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.867489 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:34.868390 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:34.868393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.869011 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.869012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:34.869071 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:34.869074 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:47:34.869153 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:47:34.869193 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":17,"thread_start_us":64,"threads_started":1}
I20260926 20:47:34.869659 19935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debb3d070d9241e19adaf1ab24eef42f" candidate_uuid: "102b3ca0317848c4afaa2c46825a60c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a742844d0f42fea77730ffe165f631" is_pre_election: true
I20260926 20:47:34.869670 19801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debb3d070d9241e19adaf1ab24eef42f" candidate_uuid: "102b3ca0317848c4afaa2c46825a60c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e03f8c8b5541abbd799ce17578ccf1" is_pre_election: true
I20260926 20:47:34.869819 19801 raft_consensus.cc:2454] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 102b3ca0317848c4afaa2c46825a60c4 in term 0.
I20260926 20:47:34.869819 19935 raft_consensus.cc:2454] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 102b3ca0317848c4afaa2c46825a60c4 in term 0.
I20260926 20:47:34.870042 19602 leader_election.cc:302] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 102b3ca0317848c4afaa2c46825a60c4, 78e03f8c8b5541abbd799ce17578ccf1; no voters:
I20260926 20:47:34.870208 20006 raft_consensus.cc:2781] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:47:34.870285 20006 raft_consensus.cc:482] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:47:34.870316 20006 raft_consensus.cc:3037] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:34.871055 20006 raft_consensus.cc:504] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e03f8c8b5541abbd799ce17578ccf1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 40639 } } peers { permanent_uuid: "00a742844d0f42fea77730ffe165f631" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38599 } } peers { permanent_uuid: "102b3ca0317848c4afaa2c46825a60c4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38107 } }
I20260926 20:47:34.871212 20006 leader_election.cc:288] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 [CANDIDATE]: Term 1 election: Requested vote from peers 78e03f8c8b5541abbd799ce17578ccf1 (127.0.98.66:40639), 00a742844d0f42fea77730ffe165f631 (127.0.98.67:38599)
I20260926 20:47:34.871400 19801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debb3d070d9241e19adaf1ab24eef42f" candidate_uuid: "102b3ca0317848c4afaa2c46825a60c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e03f8c8b5541abbd799ce17578ccf1"
I20260926 20:47:34.871440 19935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debb3d070d9241e19adaf1ab24eef42f" candidate_uuid: "102b3ca0317848c4afaa2c46825a60c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a742844d0f42fea77730ffe165f631"
I20260926 20:47:34.871502 19801 raft_consensus.cc:3037] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:34.871526 19935 raft_consensus.cc:3037] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:34.872427 19935 raft_consensus.cc:2454] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 102b3ca0317848c4afaa2c46825a60c4 in term 1.
I20260926 20:47:34.872424 19801 raft_consensus.cc:2454] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 102b3ca0317848c4afaa2c46825a60c4 in term 1.
I20260926 20:47:34.872617 19601 leader_election.cc:302] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a742844d0f42fea77730ffe165f631, 102b3ca0317848c4afaa2c46825a60c4; no voters:
I20260926 20:47:34.872752 20006 raft_consensus.cc:2781] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:34.872931 20006 raft_consensus.cc:686] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 [term 1 LEADER]: Becoming Leader. State: Replica: 102b3ca0317848c4afaa2c46825a60c4, State: Running, Role: LEADER
I20260926 20:47:34.873032 20006 consensus_queue.cc:237] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e03f8c8b5541abbd799ce17578ccf1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 40639 } } peers { permanent_uuid: "00a742844d0f42fea77730ffe165f631" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38599 } } peers { permanent_uuid: "102b3ca0317848c4afaa2c46825a60c4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38107 } }
I20260926 20:47:34.873881 19521 catalog_manager.cc:5495] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 102b3ca0317848c4afaa2c46825a60c4 (127.0.98.65). New cstate: current_term: 1 leader_uuid: "102b3ca0317848c4afaa2c46825a60c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e03f8c8b5541abbd799ce17578ccf1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 40639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00a742844d0f42fea77730ffe165f631" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "102b3ca0317848c4afaa2c46825a60c4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38107 } health_report { overall_health: HEALTHY } } }
I20260926 20:47:34.896778 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:34.894333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34818 (local address 127.0.98.65:38107)
0926 20:47:34.894392 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:34.894394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:34.894405 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:47:34.894451 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:34.894452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:34.894474 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:34.894562 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:47:34.894668 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.895144 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.896012 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:34.896221 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:34.896303 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:34.896445 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:47:34.896560 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:47:34.896655 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:34.896674 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 20:47:34.896693 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:47:34.898680 19801 raft_consensus.cc:1261] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1 [term 1 FOLLOWER]: Refusing update from remote peer 102b3ca0317848c4afaa2c46825a60c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:47:34.899053 19935 raft_consensus.cc:1261] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631 [term 1 FOLLOWER]: Refusing update from remote peer 102b3ca0317848c4afaa2c46825a60c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:47:34.899037 20009 consensus_queue.cc:1035] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e03f8c8b5541abbd799ce17578ccf1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 40639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:47:34.899329 20009 consensus_queue.cc:1035] T debb3d070d9241e19adaf1ab24eef42f P 102b3ca0317848c4afaa2c46825a60c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a742844d0f42fea77730ffe165f631" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:47:34.901949 20015 mvcc.cc:205] Tried to move back new op lower bound from 7333706362461904896 to 7333706362360639488. Current Snapshot: MvccSnapshot[applied={T|T < 7333706362461904896}]
I20260926 20:47:34.902729 20013 mvcc.cc:205] Tried to move back new op lower bound from 7333706362461904896 to 7333706362360639488. Current Snapshot: MvccSnapshot[applied={T|T < 7333706362461904896}]
I20260926 20:47:34.904085 19521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:47:34.904176 19521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:47:34.904944 19521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2084f5310e5340d0a30e0daf463cb55e: Sending DeleteTablet for 3 replicas of tablet debb3d070d9241e19adaf1ab24eef42f
I20260926 20:47:34.905233 19781 tablet_service.cc:1514] Processing DeleteTablet for tablet debb3d070d9241e19adaf1ab24eef42f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:47:34 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:46182
I20260926 20:47:34.905265 19915 tablet_service.cc:1514] Processing DeleteTablet for tablet debb3d070d9241e19adaf1ab24eef42f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:47:34 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:50220
I20260926 20:47:34.905364 19647 tablet_service.cc:1514] Processing DeleteTablet for tablet debb3d070d9241e19adaf1ab24eef42f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:47:34 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:34806
I20260926 20:47:34.905475 20022 tablet_replica.cc:330] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1: stopping tablet replica
I20260926 20:47:34.905578 20022 raft_consensus.cc:2229] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:47:34.905604 20023 tablet_replica.cc:330] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631: stopping tablet replica
I20260926 20:47:34.905743 20022 raft_consensus.cc:2258] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:47:34.905758 20023 raft_consensus.cc:2229] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:47:34.905834 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 19584
I20260926 20:47:34.908492 20022 ts_tablet_manager.cc:1905] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:47:34.910218 20022 ts_tablet_manager.cc:1918] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:47:34.910292 20022 log.cc:1196] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/debb3d070d9241e19adaf1ab24eef42f
I20260926 20:47:34.910594 20022 ts_tablet_manager.cc:1939] T debb3d070d9241e19adaf1ab24eef42f P 78e03f8c8b5541abbd799ce17578ccf1: Deleting consensus metadata
I20260926 20:47:34.911080 19508 catalog_manager.cc:4841] TS 78e03f8c8b5541abbd799ce17578ccf1 (127.0.98.66:40639): tablet debb3d070d9241e19adaf1ab24eef42f (table test-table [id=ca2af901be974eb9998cc23f287c469f]) successfully deleted
W20260926 20:47:34.912381 19507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:38107 (error 108)
I20260926 20:47:34.912849 20023 raft_consensus.cc:2258] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 20:47:34.913111 19995 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:47:34.912764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:38107 (local address 127.0.0.1:34822)
0926 20:47:34.912850 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:34.912863 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:34.912912 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:34.913065 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:38107: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 20:47:34.913213 19507 catalog_manager.cc:4584] TS 102b3ca0317848c4afaa2c46825a60c4 (127.0.98.65:38107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet debb3d070d9241e19adaf1ab24eef42f: Network error: Client connection negotiation failed: client connection to 127.0.98.65:38107: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:47:34.913355 20023 ts_tablet_manager.cc:1905] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:47:34.913516 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 19718
I20260926 20:47:34.915061 20023 ts_tablet_manager.cc:1918] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:47:34.915153 20023 log.cc:1196] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/debb3d070d9241e19adaf1ab24eef42f
I20260926 20:47:34.915510 20023 ts_tablet_manager.cc:1939] T debb3d070d9241e19adaf1ab24eef42f P 00a742844d0f42fea77730ffe165f631: Deleting consensus metadata
I20260926 20:47:34.915983 19507 catalog_manager.cc:4841] TS 00a742844d0f42fea77730ffe165f631 (127.0.98.67:38599): tablet debb3d070d9241e19adaf1ab24eef42f (table test-table [id=ca2af901be974eb9998cc23f287c469f]) successfully deleted
I20260926 20:47:34.920342 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 19852
I20260926 20:47:34.925278 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 19490
2026-09-26T20:47:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2207 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T20:47:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:34Z Disabled control of system clock
I20260926 20:47:34.954453 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40419
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:34581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:40419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:47:35.036343 20029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:35.036728 20029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:35.036770 20029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:35.038973 20029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:35.039038 20029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:35.039059 20029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:35.039078 20029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:35.039098 20029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:35.041600 20029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:40419
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:35.041903 20029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:35.042320 20029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:35.047821 20037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:35.047828 20035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:35.047961 20034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:35.047955 20029 server_base.cc:1034] running on GCE node
I20260926 20:47:35.048369 20029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:35.048713 20029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:35.049909 20029 hybrid_clock.cc:648] HybridClock initialized: now 1790455655049894 us; error 34 us; skew 500 ppm
I20260926 20:47:35.051463 20029 webserver.cc:460] Webserver started at http://127.0.98.126:38335/ using document root <none> and password file <none>
I20260926 20:47:35.051796 20029 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:35.051918 20029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:35.052083 20029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:35.053604 20029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "102624cab0f045d2941818b8dc66ac41"
format_stamp: "Formatted at 2026-09-26 20:47:35 on dist-test-slave-4hd3"
server_key: "7edd2ecd4394ae88cf5ca48401235516"
server_key_iv: "0590e2dc3028ad4e0524e2397ff4cc7d"
server_key_version: "encryptionkey@0"
I20260926 20:47:35.053985 20029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "102624cab0f045d2941818b8dc66ac41"
format_stamp: "Formatted at 2026-09-26 20:47:35 on dist-test-slave-4hd3"
server_key: "7edd2ecd4394ae88cf5ca48401235516"
server_key_iv: "0590e2dc3028ad4e0524e2397ff4cc7d"
server_key_version: "encryptionkey@0"
I20260926 20:47:35.055258 20029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:47:35.056118 20043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:35.056325 20029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:35.056463 20029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "102624cab0f045d2941818b8dc66ac41"
format_stamp: "Formatted at 2026-09-26 20:47:35 on dist-test-slave-4hd3"
server_key: "7edd2ecd4394ae88cf5ca48401235516"
server_key_iv: "0590e2dc3028ad4e0524e2397ff4cc7d"
server_key_version: "encryptionkey@0"
I20260926 20:47:35.056546 20029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:35.070343 20029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:35.070693 20029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:35.070853 20029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:35.075414 20029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:40419
I20260926 20:47:35.075474 20095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:40419 every 8 connection(s)
I20260926 20:47:35.075903 20029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:35.077162 20096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:35.079020 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 20029
I20260926 20:47:35.079175 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:35.079386 393 external_mini_cluster.cc:1428] Setting key 54f704e769be84a2e5768eae2b097f3c
I20260926 20:47:35.080281 20096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41: Bootstrap starting.
I20260926 20:47:35.080886 20096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:35.081231 20096 log.cc:824] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:35.081984 20096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41: No bootstrap required, opened a new log
I20260926 20:47:35.084312 20096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "102624cab0f045d2941818b8dc66ac41" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } }
I20260926 20:47:35.084476 20096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:35.084502 20096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 102624cab0f045d2941818b8dc66ac41, State: Initialized, Role: FOLLOWER
I20260926 20:47:35.084646 20096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "102624cab0f045d2941818b8dc66ac41" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } }
I20260926 20:47:35.084744 20096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:35.084777 20096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:35.084826 20096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:35.085486 20096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "102624cab0f045d2941818b8dc66ac41" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } }
I20260926 20:47:35.085608 20096 leader_election.cc:302] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 102624cab0f045d2941818b8dc66ac41; no voters:
I20260926 20:47:35.085659 20099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.080228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.0.98.126:40419)
0926 20:47:35.080486 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:35.080495 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:35.080521 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:47:35.080949 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:35.080958 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:35.081043 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:35.081156 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:35.081403 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.082488 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.083567 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.084022 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.084112 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:35.084892 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:35.084900 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:35.084901 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:35.085104 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:35.085138 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:35.085149 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:35.085202 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:35.085317 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 20:47:35.085394 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20260926 20:47:35.085814 20096 leader_election.cc:288] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:35.085983 20102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:35.086062 20096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:35.086115 20102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41 [term 1 LEADER]: Becoming Leader. State: Replica: 102624cab0f045d2941818b8dc66ac41, State: Running, Role: LEADER
I20260926 20:47:35.086186 20102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "102624cab0f045d2941818b8dc66ac41" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } }
I20260926 20:47:35.086585 20102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 102624cab0f045d2941818b8dc66ac41. Latest consensus state: current_term: 1 leader_uuid: "102624cab0f045d2941818b8dc66ac41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "102624cab0f045d2941818b8dc66ac41" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } }
I20260926 20:47:35.086596 20103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "102624cab0f045d2941818b8dc66ac41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "102624cab0f045d2941818b8dc66ac41" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } }
I20260926 20:47:35.086778 20102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:35.086824 20103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:35.087229 20110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:35.087751 20110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:35.089316 20110 catalog_manager.cc:1347] Generated new cluster ID: ea67c577f3c545c6a7a2b70173a53cab
I20260926 20:47:35.089381 20110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:47:35.094254 20110 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:35.094936 20110 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:35.097285 20110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41: Generated new TSK 0
I20260926 20:47:35.097440 20110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:35.100814 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40419
--builtin_ntp_servers=127.0.98.84:34581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:47:35.181293 20120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:35.181553 20120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:35.181592 20120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:35.182773 20120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:35.182843 20120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:35.182883 20120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260926 20:47:35.184262 20120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:35.184549 20120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:35.184849 20120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:35.190196 20126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:35.190178 20125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:35.190176 20128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:35.190845 20120 server_base.cc:1034] running on GCE node
I20260926 20:47:35.191074 20120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:35.191325 20120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:35.192509 20120 hybrid_clock.cc:648] HybridClock initialized: now 1790455655192489 us; error 35 us; skew 500 ppm
I20260926 20:47:35.193848 20120 webserver.cc:460] Webserver started at http://127.0.98.65:38517/ using document root <none> and password file <none>
I20260926 20:47:35.194120 20120 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:35.194195 20120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:35.194345 20120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:35.195447 20120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-0/data/instance:
uuid: "14d722e2eaac4b7083c874de23fedb4b"
format_stamp: "Formatted at 2026-09-26 20:47:35 on dist-test-slave-4hd3"
server_key: "ff0a98a5edfce5f7b160b68f19beb03b"
server_key_iv: "12c49566ec7b75446063f1992ea2fb05"
server_key_version: "encryptionkey@0"
I20260926 20:47:35.195869 20120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-0/wal/instance:
uuid: "14d722e2eaac4b7083c874de23fedb4b"
format_stamp: "Formatted at 2026-09-26 20:47:35 on dist-test-slave-4hd3"
server_key: "ff0a98a5edfce5f7b160b68f19beb03b"
server_key_iv: "12c49566ec7b75446063f1992ea2fb05"
server_key_version: "encryptionkey@0"
I20260926 20:47:35.197263 20120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:47:35.198076 20134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:35.198222 20120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:47:35.198340 20120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-0/wal
uuid: "14d722e2eaac4b7083c874de23fedb4b"
format_stamp: "Formatted at 2026-09-26 20:47:35 on dist-test-slave-4hd3"
server_key: "ff0a98a5edfce5f7b160b68f19beb03b"
server_key_iv: "12c49566ec7b75446063f1992ea2fb05"
server_key_version: "encryptionkey@0"
I20260926 20:47:35.198416 20120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:35.220117 20120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:35.220443 20120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:35.220602 20120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:35.220844 20120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:35.221191 20120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:35.221227 20120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:35.221251 20120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:35.221262 20120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:35.227047 20120 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:46237
I20260926 20:47:35.227142 20247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:46237 every 8 connection(s)
I20260926 20:47:35.227453 20120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-0/data/info.pb
I20260926 20:47:35.232436 20099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.228220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:47027 (local address 127.0.98.126:40419)
0926 20:47:35.228262 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:35.228265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:35.228508 (+ 243us) server_negotiation.cc:407] Connection header received
0926 20:47:35.228988 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:35.228989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:35.229000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:35.229050 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:35.229685 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.230171 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.231128 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.231310 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.231367 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:35.232199 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:35.232200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:35.232200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:35.232205 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:35.232217 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:35.232221 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:35.232265 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:35.232326 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 20:47:35.232346 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:47:35.232620 20250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.228215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40419 (local address 127.0.98.65:47027)
0926 20:47:35.228381 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:35.228416 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:35.228878 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:35.229110 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:35.229114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:35.229121 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:35.229611 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:35.229615 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.230249 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.230250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:35.231047 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:35.231049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.231384 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.231385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:35.231489 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:35.232076 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:47:35.232076 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:35.232104 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:47:35.232126 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:35.232265 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:35.232267 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:35.232270 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:47:35.232317 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 20:47:35.232396 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":89,"threads_started":1}
I20260926 20:47:35.233215 20248 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40419
I20260926 20:47:35.233304 20248 heartbeater.cc:464] Registering TS with master...
I20260926 20:47:35.233518 20248 heartbeater.cc:511] Master 127.0.98.126:40419 requested a full tablet report, sending...
I20260926 20:47:35.234038 20052 ts_manager.cc:194] Registered new tserver with Master: 14d722e2eaac4b7083c874de23fedb4b (127.0.98.65:46237)
I20260926 20:47:35.234625 20052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.65:47027
I20260926 20:47:35.235869 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 20120
I20260926 20:47:35.235982 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-0/wal/instance
I20260926 20:47:35.236179 393 external_mini_cluster.cc:1428] Setting key d520b28fc7d6cfdd9b4a9ca533949a11
I20260926 20:47:35.237056 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40419
--builtin_ntp_servers=127.0.98.84:34581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:47:35.318548 20251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:35.318742 20251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:35.318763 20251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:35.319965 20251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:35.320008 20251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:35.320034 20251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260926 20:47:35.321357 20251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:35.321620 20251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:35.321913 20251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:35.326843 20259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:35.326845 20256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:35.326967 20251 server_base.cc:1034] running on GCE node
W20260926 20:47:35.327103 20257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:35.327391 20251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:35.327665 20251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:35.328852 20251 hybrid_clock.cc:648] HybridClock initialized: now 1790455655328845 us; error 30 us; skew 500 ppm
I20260926 20:47:35.330112 20251 webserver.cc:460] Webserver started at http://127.0.98.66:37021/ using document root <none> and password file <none>
I20260926 20:47:35.330363 20251 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:35.330435 20251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:35.330602 20251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:35.331661 20251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-1/data/instance:
uuid: "0496df265d174bd2953d92c31e0c2853"
format_stamp: "Formatted at 2026-09-26 20:47:35 on dist-test-slave-4hd3"
server_key: "ec448cd36bcf0b799355659c0fea398e"
server_key_iv: "9587ea3fac82fe26f69a2bf507db46b3"
server_key_version: "encryptionkey@0"
I20260926 20:47:35.332089 20251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-1/wal/instance:
uuid: "0496df265d174bd2953d92c31e0c2853"
format_stamp: "Formatted at 2026-09-26 20:47:35 on dist-test-slave-4hd3"
server_key: "ec448cd36bcf0b799355659c0fea398e"
server_key_iv: "9587ea3fac82fe26f69a2bf507db46b3"
server_key_version: "encryptionkey@0"
I20260926 20:47:35.333387 20251 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:35.334106 20265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:35.334270 20251 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:47:35.334370 20251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-1/wal
uuid: "0496df265d174bd2953d92c31e0c2853"
format_stamp: "Formatted at 2026-09-26 20:47:35 on dist-test-slave-4hd3"
server_key: "ec448cd36bcf0b799355659c0fea398e"
server_key_iv: "9587ea3fac82fe26f69a2bf507db46b3"
server_key_version: "encryptionkey@0"
I20260926 20:47:35.334460 20251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:35.348528 20251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:35.348852 20251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:35.349014 20251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:35.349251 20251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:35.349588 20251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:35.349642 20251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:35.349679 20251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:35.349707 20251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:35.355078 20251 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:43517
I20260926 20:47:35.355159 20378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:43517 every 8 connection(s)
I20260926 20:47:35.355506 20251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-1/data/info.pb
I20260926 20:47:35.361119 20099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.356316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:59393 (local address 127.0.98.126:40419)
0926 20:47:35.356409 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:35.356412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:35.356628 (+ 216us) server_negotiation.cc:407] Connection header received
0926 20:47:35.357142 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:35.357144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:35.357155 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:35.357215 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:35.357907 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.358498 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.359701 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.359958 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.360017 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:35.360902 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:35.360904 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:35.360904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:35.360908 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:35.360920 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:35.360923 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:35.360966 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:35.361032 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 20:47:35.361051 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:47:35.361331 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.356290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40419 (local address 127.0.98.66:59393)
0926 20:47:35.356500 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:35.356534 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:35.357024 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:35.357294 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:35.357298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:35.357306 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:35.357819 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:35.357823 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.358650 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.358651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:35.359615 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:35.359618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.360038 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.360039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:35.360148 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:35.360767 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:47:35.360768 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:35.360796 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:47:35.360820 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:35.360966 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:35.360969 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:35.360971 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:47:35.361020 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 20:47:35.361095 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":111,"threads_started":1}
I20260926 20:47:35.361703 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 20251
I20260926 20:47:35.361819 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-1/wal/instance
I20260926 20:47:35.362059 393 external_mini_cluster.cc:1428] Setting key c66ea6f941e52153b97f4fb625c013a4
I20260926 20:47:35.362068 20379 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40419
I20260926 20:47:35.362161 20379 heartbeater.cc:464] Registering TS with master...
I20260926 20:47:35.362471 20379 heartbeater.cc:511] Master 127.0.98.126:40419 requested a full tablet report, sending...
I20260926 20:47:35.362874 20052 ts_manager.cc:194] Registered new tserver with Master: 0496df265d174bd2953d92c31e0c2853 (127.0.98.66:43517)
I20260926 20:47:35.362986 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40419
--builtin_ntp_servers=127.0.98.84:34581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:47:35.363368 20052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.66:59393
W20260926 20:47:35.442585 20382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:35.442781 20382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:35.442801 20382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:35.443871 20382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:35.443902 20382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:35.443924 20382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260926 20:47:35.445176 20382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:35.445359 20382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:35.445595 20382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:35.450598 20388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:35.450750 20382 server_base.cc:1034] running on GCE node
W20260926 20:47:35.450590 20387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:35.450618 20390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:35.451074 20382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:35.451334 20382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:35.452502 20382 hybrid_clock.cc:648] HybridClock initialized: now 1790455655452493 us; error 33 us; skew 500 ppm
I20260926 20:47:35.453717 20382 webserver.cc:460] Webserver started at http://127.0.98.67:41649/ using document root <none> and password file <none>
I20260926 20:47:35.453965 20382 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:35.454041 20382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:35.454188 20382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:35.455224 20382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-2/data/instance:
uuid: "d7e0132195bd4d8989b152021fb688f9"
format_stamp: "Formatted at 2026-09-26 20:47:35 on dist-test-slave-4hd3"
server_key: "0cea03cac839aefb1711ad13336f9a54"
server_key_iv: "bc0382458a75c9f2182dbff13e5ca716"
server_key_version: "encryptionkey@0"
I20260926 20:47:35.455585 20382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-2/wal/instance:
uuid: "d7e0132195bd4d8989b152021fb688f9"
format_stamp: "Formatted at 2026-09-26 20:47:35 on dist-test-slave-4hd3"
server_key: "0cea03cac839aefb1711ad13336f9a54"
server_key_iv: "bc0382458a75c9f2182dbff13e5ca716"
server_key_version: "encryptionkey@0"
I20260926 20:47:35.456861 20382 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:47:35.457674 20396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:35.457861 20382 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:47:35.457960 20382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-2/wal
uuid: "d7e0132195bd4d8989b152021fb688f9"
format_stamp: "Formatted at 2026-09-26 20:47:35 on dist-test-slave-4hd3"
server_key: "0cea03cac839aefb1711ad13336f9a54"
server_key_iv: "bc0382458a75c9f2182dbff13e5ca716"
server_key_version: "encryptionkey@0"
I20260926 20:47:35.458042 20382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:35.473593 20382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:35.473919 20382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:35.474068 20382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:35.474294 20382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:47:35.474627 20382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:47:35.474681 20382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:35.474722 20382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:47:35.474749 20382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:35.480656 20382 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:34313
I20260926 20:47:35.480733 20509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:34313 every 8 connection(s)
I20260926 20:47:35.481071 20382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-2/data/info.pb
I20260926 20:47:35.486325 20099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.481811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:60109 (local address 127.0.98.126:40419)
0926 20:47:35.481883 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:35.481885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:35.482148 (+ 263us) server_negotiation.cc:407] Connection header received
0926 20:47:35.482614 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:35.482615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:35.482625 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:35.482691 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:35.483421 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.484014 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.484939 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.485118 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.485172 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:35.486097 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:35.486098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:35.486099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:35.486104 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:35.486117 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:35.486121 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:35.486174 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:35.486248 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 20:47:35.486267 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:47:35.486497 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.481799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40419 (local address 127.0.98.67:60109)
0926 20:47:35.481999 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:35.482040 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:35.482515 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:35.482745 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:35.482748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:35.482757 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:47:35.483311 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:35.483316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.484094 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.484095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:35.484854 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:35.484856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.485191 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.485191 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:35.485292 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:35.485911 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:47:35.485912 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:47:35.485953 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:47:35.486024 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:47:35.486174 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:47:35.486178 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:47:35.486181 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:47:35.486237 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 20:47:35.486313 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260926 20:47:35.486922 20510 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40419
I20260926 20:47:35.487051 20510 heartbeater.cc:464] Registering TS with master...
I20260926 20:47:35.487263 20510 heartbeater.cc:511] Master 127.0.98.126:40419 requested a full tablet report, sending...
I20260926 20:47:35.487601 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 20382
I20260926 20:47:35.487645 20052 ts_manager.cc:194] Registered new tserver with Master: d7e0132195bd4d8989b152021fb688f9 (127.0.98.67:34313)
I20260926 20:47:35.487715 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-2/wal/instance
I20260926 20:47:35.487926 393 external_mini_cluster.cc:1428] Setting key 26c029e0e21384d13d3b87391945b07e
I20260926 20:47:35.488088 20052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.67:60109
I20260926 20:47:35.488801 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:47:35.492672 20099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.490106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56744 (local address 127.0.98.126:40419)
0926 20:47:35.490157 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:35.490158 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:35.490254 (+ 96us) server_negotiation.cc:407] Connection header received
0926 20:47:35.490309 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:35.490310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:35.490318 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:35.490369 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:35.490485 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.490920 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.491597 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.491838 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.491892 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:35.492524 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:35.492525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:35.492525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:35.492529 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:35.492538 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:35.492541 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:35.492578 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:35.492611 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 20:47:35.492626 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 20:47:35.494093 20052 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:47:35.495546 20052 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:47:35.501286 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.497753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33626 (local address 127.0.98.66:43517)
0926 20:47:35.497955 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:35.497959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:35.497972 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:47:35.498026 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:35.498028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:35.498044 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:35.498130 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:35.498309 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.499195 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.500263 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.500859 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.500965 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:35.501060 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:35.501167 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 20:47:35.501203 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
I20260926 20:47:35.501276 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.497565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:43517 (local address 127.0.0.1:33626)
0926 20:47:35.497684 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:35.497722 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:35.497787 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:35.498147 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:35.498151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:35.498152 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:35.498221 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:35.498223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.499279 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.499280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:35.500163 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:35.500166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.500968 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.500969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:35.501066 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:35.501080 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 20:47:35.501153 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 20:47:35.501203 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":82,"threads_started":1}
I20260926 20:47:35.502074 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.497583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:34313 (local address 127.0.0.1:46590)
0926 20:47:35.498193 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:35.498203 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:35.498251 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:35.498584 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:35.498585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:35.498587 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:35.498621 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:35.498622 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.499808 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.499808 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:35.500978 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:35.500980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.501834 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.501835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:35.501909 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:35.501913 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:47:35.501972 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 20:47:35.502008 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260926 20:47:35.502115 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.497790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46590 (local address 127.0.98.67:34313)
0926 20:47:35.497956 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:35.497960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:35.498249 (+ 289us) server_negotiation.cc:407] Connection header received
0926 20:47:35.498358 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:35.498361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:35.498377 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:35.498524 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:35.498685 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.499700 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.501058 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.501749 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.501824 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:35.501907 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:35.501982 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 20:47:35.502021 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260926 20:47:35.502198 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.497567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:46237 (local address 127.0.0.1:40934)
0926 20:47:35.498026 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:35.498038 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:35.498090 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:35.498355 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:35.498356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:35.498358 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:35.498427 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:35.498429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.499745 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.499746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:35.500924 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:35.500927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.501944 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.501945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:35.502020 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:35.502024 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:47:35.502075 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 20:47:35.502118 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":34,"thread_start_us":73,"threads_started":1}
I20260926 20:47:35.502362 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.497812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.0.98.65:46237)
0926 20:47:35.498000 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:35.498003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:35.498090 (+ 87us) server_negotiation.cc:407] Connection header received
0926 20:47:35.498176 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:35.498178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:35.498191 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:35.498261 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:35.498512 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.499631 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.501033 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.501764 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.501824 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:35.501907 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:35.502259 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 20:47:35.502305 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":103,"threads_started":1}
I20260926 20:47:35.502308 20313 tablet_service.cc:1467] Processing CreateTablet for tablet c1d97d9e20004f21815810ea761aff56 (DEFAULT_TABLE table=test-table [id=eaa40ec93911405e870188fff8b05f26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:47:35.502704 20313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1d97d9e20004f21815810ea761aff56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:35.502981 20444 tablet_service.cc:1467] Processing CreateTablet for tablet c1d97d9e20004f21815810ea761aff56 (DEFAULT_TABLE table=test-table [id=eaa40ec93911405e870188fff8b05f26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:47:35.503274 20444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1d97d9e20004f21815810ea761aff56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:35.505460 20524 tablet_bootstrap.cc:492] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853: Bootstrap starting.
I20260926 20:47:35.506027 20182 tablet_service.cc:1467] Processing CreateTablet for tablet c1d97d9e20004f21815810ea761aff56 (DEFAULT_TABLE table=test-table [id=eaa40ec93911405e870188fff8b05f26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:47:35.506178 20524 tablet_bootstrap.cc:654] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:35.506361 20182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1d97d9e20004f21815810ea761aff56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:35.506480 20524 log.cc:824] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:35.506884 20525 tablet_bootstrap.cc:492] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9: Bootstrap starting.
I20260926 20:47:35.507513 20525 tablet_bootstrap.cc:654] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:35.507866 20525 log.cc:824] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:35.508287 20524 tablet_bootstrap.cc:492] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853: No bootstrap required, opened a new log
I20260926 20:47:35.508349 20524 ts_tablet_manager.cc:1397] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 20:47:35.508621 20525 tablet_bootstrap.cc:492] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9: No bootstrap required, opened a new log
I20260926 20:47:35.508680 20525 ts_tablet_manager.cc:1397] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:47:35.509783 20528 tablet_bootstrap.cc:492] T c1d97d9e20004f21815810ea761aff56 P 14d722e2eaac4b7083c874de23fedb4b: Bootstrap starting.
I20260926 20:47:35.510396 20528 tablet_bootstrap.cc:654] T c1d97d9e20004f21815810ea761aff56 P 14d722e2eaac4b7083c874de23fedb4b: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:35.510465 20524 raft_consensus.cc:348] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0496df265d174bd2953d92c31e0c2853" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43517 } } peers { permanent_uuid: "14d722e2eaac4b7083c874de23fedb4b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46237 } } peers { permanent_uuid: "d7e0132195bd4d8989b152021fb688f9" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34313 } }
I20260926 20:47:35.510628 20524 raft_consensus.cc:374] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:35.510663 20524 raft_consensus.cc:729] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0496df265d174bd2953d92c31e0c2853, State: Initialized, Role: FOLLOWER
I20260926 20:47:35.510689 20528 log.cc:824] T c1d97d9e20004f21815810ea761aff56 P 14d722e2eaac4b7083c874de23fedb4b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:35.510780 20524 consensus_queue.cc:260] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0496df265d174bd2953d92c31e0c2853" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43517 } } peers { permanent_uuid: "14d722e2eaac4b7083c874de23fedb4b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46237 } } peers { permanent_uuid: "d7e0132195bd4d8989b152021fb688f9" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34313 } }
I20260926 20:47:35.511093 20524 ts_tablet_manager.cc:1428] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:47:35.511122 20379 heartbeater.cc:503] Master 127.0.98.126:40419 was elected leader, sending a full tablet report...
I20260926 20:47:35.511437 20528 tablet_bootstrap.cc:492] T c1d97d9e20004f21815810ea761aff56 P 14d722e2eaac4b7083c874de23fedb4b: No bootstrap required, opened a new log
I20260926 20:47:35.511504 20528 ts_tablet_manager.cc:1397] T c1d97d9e20004f21815810ea761aff56 P 14d722e2eaac4b7083c874de23fedb4b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:47:35.511391 20525 raft_consensus.cc:348] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0496df265d174bd2953d92c31e0c2853" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43517 } } peers { permanent_uuid: "14d722e2eaac4b7083c874de23fedb4b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46237 } } peers { permanent_uuid: "d7e0132195bd4d8989b152021fb688f9" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34313 } }
I20260926 20:47:35.511556 20525 raft_consensus.cc:374] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:35.511600 20525 raft_consensus.cc:729] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7e0132195bd4d8989b152021fb688f9, State: Initialized, Role: FOLLOWER
I20260926 20:47:35.511708 20525 consensus_queue.cc:260] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0496df265d174bd2953d92c31e0c2853" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43517 } } peers { permanent_uuid: "14d722e2eaac4b7083c874de23fedb4b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46237 } } peers { permanent_uuid: "d7e0132195bd4d8989b152021fb688f9" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34313 } }
I20260926 20:47:35.512004 20525 ts_tablet_manager.cc:1428] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:47:35.512130 20510 heartbeater.cc:503] Master 127.0.98.126:40419 was elected leader, sending a full tablet report...
I20260926 20:47:35.513736 20528 raft_consensus.cc:348] T c1d97d9e20004f21815810ea761aff56 P 14d722e2eaac4b7083c874de23fedb4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0496df265d174bd2953d92c31e0c2853" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43517 } } peers { permanent_uuid: "14d722e2eaac4b7083c874de23fedb4b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46237 } } peers { permanent_uuid: "d7e0132195bd4d8989b152021fb688f9" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34313 } }
I20260926 20:47:35.513876 20528 raft_consensus.cc:374] T c1d97d9e20004f21815810ea761aff56 P 14d722e2eaac4b7083c874de23fedb4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:35.513921 20528 raft_consensus.cc:729] T c1d97d9e20004f21815810ea761aff56 P 14d722e2eaac4b7083c874de23fedb4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d722e2eaac4b7083c874de23fedb4b, State: Initialized, Role: FOLLOWER
I20260926 20:47:35.514030 20528 consensus_queue.cc:260] T c1d97d9e20004f21815810ea761aff56 P 14d722e2eaac4b7083c874de23fedb4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0496df265d174bd2953d92c31e0c2853" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43517 } } peers { permanent_uuid: "14d722e2eaac4b7083c874de23fedb4b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46237 } } peers { permanent_uuid: "d7e0132195bd4d8989b152021fb688f9" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34313 } }
I20260926 20:47:35.514303 20528 ts_tablet_manager.cc:1428] T c1d97d9e20004f21815810ea761aff56 P 14d722e2eaac4b7083c874de23fedb4b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:47:35.514420 20248 heartbeater.cc:503] Master 127.0.98.126:40419 was elected leader, sending a full tablet report...
W20260926 20:47:35.606140 20380 tablet.cc:2369] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:47:35.728166 20249 tablet.cc:2369] T c1d97d9e20004f21815810ea761aff56 P 14d722e2eaac4b7083c874de23fedb4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:47:35.731613 20511 tablet.cc:2369] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:47:35.738567 20530 raft_consensus.cc:482] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:47:35.738723 20530 raft_consensus.cc:504] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0496df265d174bd2953d92c31e0c2853" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43517 } } peers { permanent_uuid: "14d722e2eaac4b7083c874de23fedb4b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46237 } } peers { permanent_uuid: "d7e0132195bd4d8989b152021fb688f9" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34313 } }
I20260926 20:47:35.739105 20530 leader_election.cc:288] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14d722e2eaac4b7083c874de23fedb4b (127.0.98.65:46237), d7e0132195bd4d8989b152021fb688f9 (127.0.98.67:34313)
I20260926 20:47:35.742123 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.739504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:58429 (local address 127.0.98.65:46237)
0926 20:47:35.739561 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:35.739563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:35.739615 (+ 52us) server_negotiation.cc:407] Connection header received
0926 20:47:35.739663 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:35.739664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:35.739677 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:35.739785 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:35.739960 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.740427 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.741387 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.741732 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.741830 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:35.741839 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:35.742009 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 20:47:35.742046 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 20:47:35.742130 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.739370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:52003 (local address 127.0.98.67:34313)
0926 20:47:35.739413 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:35.739415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:35.739461 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:47:35.739553 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:35.739559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:35.739569 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:35.739649 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:47:35.739826 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.740393 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.741378 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.741757 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.741844 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:35.741853 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:35.742016 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 20:47:35.742056 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:47:35.742151 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.739290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:34313 (local address 127.0.98.66:52003)
0926 20:47:35.739400 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:35.739413 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:35.739469 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:35.739657 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:35.739658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:35.739660 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:35.739759 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:35.739761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.740503 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.740504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:35.741283 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:35.741285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.741845 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.741846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:35.741942 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:35.741944 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:47:35.742011 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 20:47:35.742056 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":29}
I20260926 20:47:35.742161 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.739363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:46237 (local address 127.0.98.66:58429)
0926 20:47:35.739535 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:35.739546 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:47:35.739603 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:47:35.739791 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:47:35.739792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:47:35.739794 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:47:35.739876 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:35.739878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.740512 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.740513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:35.741290 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:47:35.741292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.741832 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.741833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:47:35.741923 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:35.741926 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:47:35.742001 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:47:35.742043 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":3,"thread_start_us":106,"threads_started":1}
I20260926 20:47:35.742535 20464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1d97d9e20004f21815810ea761aff56" candidate_uuid: "0496df265d174bd2953d92c31e0c2853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7e0132195bd4d8989b152021fb688f9" is_pre_election: true
I20260926 20:47:35.742533 20202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1d97d9e20004f21815810ea761aff56" candidate_uuid: "0496df265d174bd2953d92c31e0c2853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d722e2eaac4b7083c874de23fedb4b" is_pre_election: true
I20260926 20:47:35.742695 20202 raft_consensus.cc:2454] T c1d97d9e20004f21815810ea761aff56 P 14d722e2eaac4b7083c874de23fedb4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0496df265d174bd2953d92c31e0c2853 in term 0.
I20260926 20:47:35.742695 20464 raft_consensus.cc:2454] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0496df265d174bd2953d92c31e0c2853 in term 0.
I20260926 20:47:35.742972 20268 leader_election.cc:302] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0496df265d174bd2953d92c31e0c2853, d7e0132195bd4d8989b152021fb688f9; no voters:
I20260926 20:47:35.743147 20530 raft_consensus.cc:2781] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:47:35.743186 20530 raft_consensus.cc:482] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:47:35.743201 20530 raft_consensus.cc:3037] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:35.743882 20530 raft_consensus.cc:504] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0496df265d174bd2953d92c31e0c2853" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43517 } } peers { permanent_uuid: "14d722e2eaac4b7083c874de23fedb4b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46237 } } peers { permanent_uuid: "d7e0132195bd4d8989b152021fb688f9" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34313 } }
I20260926 20:47:35.744082 20530 leader_election.cc:288] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [CANDIDATE]: Term 1 election: Requested vote from peers 14d722e2eaac4b7083c874de23fedb4b (127.0.98.65:46237), d7e0132195bd4d8989b152021fb688f9 (127.0.98.67:34313)
I20260926 20:47:35.744158 20464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1d97d9e20004f21815810ea761aff56" candidate_uuid: "0496df265d174bd2953d92c31e0c2853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7e0132195bd4d8989b152021fb688f9"
I20260926 20:47:35.744227 20202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1d97d9e20004f21815810ea761aff56" candidate_uuid: "0496df265d174bd2953d92c31e0c2853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d722e2eaac4b7083c874de23fedb4b"
I20260926 20:47:35.744248 20464 raft_consensus.cc:3037] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:35.744309 20202 raft_consensus.cc:3037] T c1d97d9e20004f21815810ea761aff56 P 14d722e2eaac4b7083c874de23fedb4b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:35.745007 20202 raft_consensus.cc:2454] T c1d97d9e20004f21815810ea761aff56 P 14d722e2eaac4b7083c874de23fedb4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0496df265d174bd2953d92c31e0c2853 in term 1.
I20260926 20:47:35.745018 20464 raft_consensus.cc:2454] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0496df265d174bd2953d92c31e0c2853 in term 1.
I20260926 20:47:35.745178 20269 leader_election.cc:302] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0496df265d174bd2953d92c31e0c2853, 14d722e2eaac4b7083c874de23fedb4b; no voters:
I20260926 20:47:35.745283 20530 raft_consensus.cc:2781] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:35.745441 20530 raft_consensus.cc:686] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [term 1 LEADER]: Becoming Leader. State: Replica: 0496df265d174bd2953d92c31e0c2853, State: Running, Role: LEADER
I20260926 20:47:35.745558 20530 consensus_queue.cc:237] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0496df265d174bd2953d92c31e0c2853" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43517 } } peers { permanent_uuid: "14d722e2eaac4b7083c874de23fedb4b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46237 } } peers { permanent_uuid: "d7e0132195bd4d8989b152021fb688f9" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34313 } }
I20260926 20:47:35.746376 20052 catalog_manager.cc:5495] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0496df265d174bd2953d92c31e0c2853 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "0496df265d174bd2953d92c31e0c2853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0496df265d174bd2953d92c31e0c2853" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d722e2eaac4b7083c874de23fedb4b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7e0132195bd4d8989b152021fb688f9" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34313 } health_report { overall_health: UNKNOWN } } }
I20260926 20:47:35.785094 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.782492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33630 (local address 127.0.98.66:43517)
0926 20:47:35.782555 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:35.782557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:35.782567 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:47:35.782678 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:35.782679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:35.782698 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:35.782772 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:47:35.782943 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.783440 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.784268 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:35.784468 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:35.784572 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:35.784737 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:47:35.784843 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:47:35.784941 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:35.784960 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 20:47:35.784977 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:47:35.787115 20202 raft_consensus.cc:1261] T c1d97d9e20004f21815810ea761aff56 P 14d722e2eaac4b7083c874de23fedb4b [term 1 FOLLOWER]: Refusing update from remote peer 0496df265d174bd2953d92c31e0c2853: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:47:35.787115 20464 raft_consensus.cc:1261] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9 [term 1 FOLLOWER]: Refusing update from remote peer 0496df265d174bd2953d92c31e0c2853: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:47:35.787580 20530 consensus_queue.cc:1035] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7e0132195bd4d8989b152021fb688f9" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:47:35.787832 20534 consensus_queue.cc:1035] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d722e2eaac4b7083c874de23fedb4b" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:47:35.790022 20537 mvcc.cc:205] Tried to move back new op lower bound from 7333706366100799488 to 7333706365934333952. Current Snapshot: MvccSnapshot[applied={T|T < 7333706366100799488}]
I20260926 20:47:35.790484 20539 mvcc.cc:205] Tried to move back new op lower bound from 7333706366100799488 to 7333706365934333952. Current Snapshot: MvccSnapshot[applied={T|T < 7333706366100799488}]
I20260926 20:47:35.790786 20538 mvcc.cc:205] Tried to move back new op lower bound from 7333706366100799488 to 7333706365934333952. Current Snapshot: MvccSnapshot[applied={T|T < 7333706366100799488}]
I20260926 20:47:35.792759 20052 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:47:35.792865 20052 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:47:35.793625 20052 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 102624cab0f045d2941818b8dc66ac41: Sending DeleteTablet for 3 replicas of tablet c1d97d9e20004f21815810ea761aff56
I20260926 20:47:35.793924 20313 tablet_service.cc:1514] Processing DeleteTablet for tablet c1d97d9e20004f21815810ea761aff56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:47:35 UTC) from {username='slave'} at 127.0.0.1:33626
I20260926 20:47:35.793988 20444 tablet_service.cc:1514] Processing DeleteTablet for tablet c1d97d9e20004f21815810ea761aff56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:47:35 UTC) from {username='slave'} at 127.0.0.1:46590
I20260926 20:47:35.794174 20547 tablet_replica.cc:330] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9: stopping tablet replica
I20260926 20:47:35.794267 20182 tablet_service.cc:1514] Processing DeleteTablet for tablet c1d97d9e20004f21815810ea761aff56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:47:35 UTC) from {username='slave'} at 127.0.0.1:40934
I20260926 20:47:35.794315 20547 raft_consensus.cc:2229] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:47:35.794487 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 20120
I20260926 20:47:35.795450 20546 tablet_replica.cc:330] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853: stopping tablet replica
I20260926 20:47:35.795552 20546 raft_consensus.cc:2229] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:47:35.795637 20546 raft_consensus.cc:2258] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:47:35.796078 20546 ts_tablet_manager.cc:1905] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:47:35.796178 20547 raft_consensus.cc:2258] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:47:35.796576 20547 ts_tablet_manager.cc:1905] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:47:35.797549 20546 ts_tablet_manager.cc:1918] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:47:35.797606 20546 log.cc:1196] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-1/wal/wals/c1d97d9e20004f21815810ea761aff56
I20260926 20:47:35.797869 20546 ts_tablet_manager.cc:1939] T c1d97d9e20004f21815810ea761aff56 P 0496df265d174bd2953d92c31e0c2853: Deleting consensus metadata
I20260926 20:47:35.798288 20045 catalog_manager.cc:4841] TS 0496df265d174bd2953d92c31e0c2853 (127.0.98.66:43517): tablet c1d97d9e20004f21815810ea761aff56 (table test-table [id=eaa40ec93911405e870188fff8b05f26]) successfully deleted
I20260926 20:47:35.798720 20547 ts_tablet_manager.cc:1918] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:47:35.798794 20547 log.cc:1196] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9: Deleting WAL directory at /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455416624047-393-0/minicluster-data/ts-2/wal/wals/c1d97d9e20004f21815810ea761aff56
I20260926 20:47:35.799046 20547 ts_tablet_manager.cc:1939] T c1d97d9e20004f21815810ea761aff56 P d7e0132195bd4d8989b152021fb688f9: Deleting consensus metadata
I20260926 20:47:35.799525 20046 catalog_manager.cc:4841] TS d7e0132195bd4d8989b152021fb688f9 (127.0.98.67:34313): tablet c1d97d9e20004f21815810ea761aff56 (table test-table [id=eaa40ec93911405e870188fff8b05f26]) successfully deleted
W20260926 20:47:35.800426 20047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:46237 (error 108)
I20260926 20:47:35.800923 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:35.800781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:46237 (local address 127.0.0.1:40944)
0926 20:47:35.800841 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 20:47:35.800883 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:46237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 20:47:35.800985 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 20251
W20260926 20:47:35.803850 20047 catalog_manager.cc:4584] TS 14d722e2eaac4b7083c874de23fedb4b (127.0.98.65:46237): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c1d97d9e20004f21815810ea761aff56: Network error: Client connection negotiation failed: client connection to 127.0.98.65:46237: connect: Connection refused (error 111)
I20260926 20:47:35.805632 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 20382
I20260926 20:47:35.810712 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 20029
2026-09-26T20:47:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (883 ms)
[----------] 6 tests from EncryptionPolicyTest (8500 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:35 dist-test-slave-4hd3 krb5kdc[20552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:35 dist-test-slave-4hd3 krb5kdc[20552](info): set up 2 sockets
Sep 26 20:47:35 dist-test-slave-4hd3 krb5kdc[20552](info): commencing operation
krb5kdc: starting...
W20260926 20:47:36.847774 393 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:36 dist-test-slave-4hd3 krb5kdc[20552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:47:36.962921 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:37.043294 20565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:37.043500 20565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:37.043524 20565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:37.044741 20565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:37.044790 20565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:37.044803 20565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:37.044816 20565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:37.044828 20565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:37.046303 20565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42411
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:37.046567 20565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:37.046840 20565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:47:37.051807 20565 server_base.cc:1034] running on GCE node
W20260926 20:47:37.051699 20570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:37.051760 20571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:37.051699 20573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:37.052304 20565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:37.052610 20565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:37.053761 20565 hybrid_clock.cc:648] HybridClock initialized: now 1790455657053746 us; error 29 us; skew 500 ppm
Sep 26 20:47:37 dist-test-slave-4hd3 krb5kdc[20552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:37.056499 20565 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:47:37.056983 20565 webserver.cc:460] Webserver started at http://127.0.0.1:34505/ using document root <none> and password file <none>
I20260926 20:47:37.057210 20565 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:37.057281 20565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:37.057437 20565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:37.058525 20565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "a2572bf3c4774c0084e4f5cfb0a6a6c6"
format_stamp: "Formatted at 2026-09-26 20:47:37 on dist-test-slave-4hd3"
server_key: "8bfd6fbc4cdc66c1404f79459115db68"
server_key_iv: "75457743978b490bf12f4d8dfa6ff36d"
server_key_version: "encryptionkey@0"
I20260926 20:47:37.058907 20565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "a2572bf3c4774c0084e4f5cfb0a6a6c6"
format_stamp: "Formatted at 2026-09-26 20:47:37 on dist-test-slave-4hd3"
server_key: "8bfd6fbc4cdc66c1404f79459115db68"
server_key_iv: "75457743978b490bf12f4d8dfa6ff36d"
server_key_version: "encryptionkey@0"
I20260926 20:47:37.060333 20565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 20:47:37.061167 20580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:37.061352 20565 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:37.061494 20565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "a2572bf3c4774c0084e4f5cfb0a6a6c6"
format_stamp: "Formatted at 2026-09-26 20:47:37 on dist-test-slave-4hd3"
server_key: "8bfd6fbc4cdc66c1404f79459115db68"
server_key_iv: "75457743978b490bf12f4d8dfa6ff36d"
server_key_version: "encryptionkey@0"
I20260926 20:47:37.061575 20565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:37.077044 20565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:37.079182 20565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:37.079344 20565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:37.083513 20565 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42411
I20260926 20:47:37.083555 20632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42411 every 8 connection(s)
I20260926 20:47:37.084013 20565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:37.085153 20633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:37.087608 20633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6: Bootstrap starting.
I20260926 20:47:37.087622 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 20565
I20260926 20:47:37.087756 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:37.087939 393 external_mini_cluster.cc:1428] Setting key a1d7459666f64ceb6a65536fbb3ff142
I20260926 20:47:37.088275 20633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:37.088613 20633 log.cc:824] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:37.089262 20633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6: No bootstrap required, opened a new log
I20260926 20:47:37.091177 20633 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2572bf3c4774c0084e4f5cfb0a6a6c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42411 } }
I20260926 20:47:37.091323 20633 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:37.091368 20633 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2572bf3c4774c0084e4f5cfb0a6a6c6, State: Initialized, Role: FOLLOWER
I20260926 20:47:37.091497 20633 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2572bf3c4774c0084e4f5cfb0a6a6c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42411 } }
I20260926 20:47:37.091595 20633 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:37.091638 20633 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:37.091688 20633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:37.092221 20633 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2572bf3c4774c0084e4f5cfb0a6a6c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42411 } }
I20260926 20:47:37.092332 20633 leader_election.cc:302] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2572bf3c4774c0084e4f5cfb0a6a6c6; no voters:
I20260926 20:47:37.092495 20633 leader_election.cc:288] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:37.092523 20638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:37.092633 20638 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6 [term 1 LEADER]: Becoming Leader. State: Replica: a2572bf3c4774c0084e4f5cfb0a6a6c6, State: Running, Role: LEADER
I20260926 20:47:37.092711 20638 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2572bf3c4774c0084e4f5cfb0a6a6c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42411 } }
I20260926 20:47:37.092741 20633 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:37.093410 20638 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2572bf3c4774c0084e4f5cfb0a6a6c6. Latest consensus state: current_term: 1 leader_uuid: "a2572bf3c4774c0084e4f5cfb0a6a6c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2572bf3c4774c0084e4f5cfb0a6a6c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42411 } } }
I20260926 20:47:37.093425 20640 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2572bf3c4774c0084e4f5cfb0a6a6c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2572bf3c4774c0084e4f5cfb0a6a6c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42411 } } }
I20260926 20:47:37.093498 20638 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:37.093537 20640 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:37.093950 20647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:37.094466 20647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 20:47:37 dist-test-slave-4hd3 krb5kdc[20552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:47:37.098816 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:37.088473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42059 (local address 127.0.0.1:42411)
0926 20:47:37.088673 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:37.088683 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:37.088716 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:47:37.089285 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:37.089293 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:37.089364 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:37.089513 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:37.090169 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:37.091141 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:37.092113 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:37.092528 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:37.092999 (+ 471us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:37.095539 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:37.095542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:37.095543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:37.095552 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:37.097320 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:37.097771 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:37.097774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:37.097774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:37.097852 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:37.098096 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:37.098097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:37.098097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:37.098305 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:37.098434 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:37.098595 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 20:47:37.098667 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
I20260926 20:47:37.099994 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:47:37.100821 20647 catalog_manager.cc:1347] Generated new cluster ID: dd8c9d38f1e84ac9926296ec42930289
I20260926 20:47:37.100885 20647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:47:37.109833 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:37.101509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57677 (local address 127.0.0.1:42411)
0926 20:47:37.101545 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:37.101547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:37.103918 (+ 2371us) server_negotiation.cc:407] Connection header received
0926 20:47:37.103989 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:37.103990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:37.104000 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:37.104043 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:37.104567 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:37.105052 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:37.105759 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:37.105998 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:37.106074 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:37.107246 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:37.107247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:37.107248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:37.107254 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:37.108681 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:37.109036 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:37.109037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:37.109038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:37.109089 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:37.109312 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:37.109313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:37.109313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:37.109422 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:37.109541 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:37.109661 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 20:47:37.109721 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:47:37.114799 20647 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:37.115294 20647 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:37.119922 20647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2572bf3c4774c0084e4f5cfb0a6a6c6: Generated new TSK 0
I20260926 20:47:37.120095 20647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:37.123205 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 20565
W20260926 20:47:37.127604 20550 connection.cc:535] client connection to 127.0.0.1:42411 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42411): Connection reset by peer (error 104)
2026-09-26T20:47:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:37 dist-test-slave-4hd3 krb5kdc[20664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:37 dist-test-slave-4hd3 krb5kdc[20664](info): set up 2 sockets
Sep 26 20:47:37 dist-test-slave-4hd3 krb5kdc[20664](info): commencing operation
krb5kdc: starting...
W20260926 20:47:38.155813 393 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:38 dist-test-slave-4hd3 krb5kdc[20664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:47:38.271420 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:38.352098 20677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:38.352341 20677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:38.352381 20677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:38.353565 20677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:38.353639 20677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:38.353662 20677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:38.353682 20677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:38.353714 20677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:38.355208 20677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40427
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:38.355490 20677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:38.355744 20677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:38.360621 20683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:38.360683 20685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:38.360700 20682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:38.360778 20677 server_base.cc:1034] running on GCE node
I20260926 20:47:38.361042 20677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:38.361351 20677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:38.362490 20677 hybrid_clock.cc:648] HybridClock initialized: now 1790455658362479 us; error 28 us; skew 500 ppm
Sep 26 20:47:38 dist-test-slave-4hd3 krb5kdc[20664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:38.369171 20677 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:47:38.369620 20677 webserver.cc:460] Webserver started at http://127.0.0.1:37187/ using document root <none> and password file <none>
I20260926 20:47:38.369838 20677 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:38.369882 20677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:38.370054 20677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:38.371124 20677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "b1268dcbc3244b08820a093d7b87ae6d"
format_stamp: "Formatted at 2026-09-26 20:47:38 on dist-test-slave-4hd3"
server_key: "33caee34838f4a783c0d2c9e55624138"
server_key_iv: "14a81556619bef331651e996fdb4f27a"
server_key_version: "encryptionkey@0"
I20260926 20:47:38.371471 20677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "b1268dcbc3244b08820a093d7b87ae6d"
format_stamp: "Formatted at 2026-09-26 20:47:38 on dist-test-slave-4hd3"
server_key: "33caee34838f4a783c0d2c9e55624138"
server_key_iv: "14a81556619bef331651e996fdb4f27a"
server_key_version: "encryptionkey@0"
I20260926 20:47:38.372850 20677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:47:38.373594 20692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:38.373839 20677 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:38.373955 20677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "b1268dcbc3244b08820a093d7b87ae6d"
format_stamp: "Formatted at 2026-09-26 20:47:38 on dist-test-slave-4hd3"
server_key: "33caee34838f4a783c0d2c9e55624138"
server_key_iv: "14a81556619bef331651e996fdb4f27a"
server_key_version: "encryptionkey@0"
I20260926 20:47:38.374050 20677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:38.394526 20677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:38.396998 20677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:38.397163 20677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:38.401036 20677 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40427
I20260926 20:47:38.401073 20744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40427 every 8 connection(s)
I20260926 20:47:38.401453 20677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:38.402323 20745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:38.404603 20745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d: Bootstrap starting.
I20260926 20:47:38.405179 20745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:38.405471 20745 log.cc:824] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:38.406177 20745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d: No bootstrap required, opened a new log
I20260926 20:47:38.406234 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 20677
I20260926 20:47:38.406347 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:38.406549 393 external_mini_cluster.cc:1428] Setting key 19e0c41ea9a56052162706b47f486b12
I20260926 20:47:38.408586 20745 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1268dcbc3244b08820a093d7b87ae6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40427 } }
I20260926 20:47:38.408769 20745 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:38.408809 20745 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1268dcbc3244b08820a093d7b87ae6d, State: Initialized, Role: FOLLOWER
I20260926 20:47:38.408955 20745 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1268dcbc3244b08820a093d7b87ae6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40427 } }
I20260926 20:47:38.409030 20745 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:38.409060 20745 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:38.409106 20745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:38.409765 20745 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1268dcbc3244b08820a093d7b87ae6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40427 } }
I20260926 20:47:38.409869 20745 leader_election.cc:302] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1268dcbc3244b08820a093d7b87ae6d; no voters:
I20260926 20:47:38.410008 20745 leader_election.cc:288] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:38.410092 20750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:38.410236 20750 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d [term 1 LEADER]: Becoming Leader. State: Replica: b1268dcbc3244b08820a093d7b87ae6d, State: Running, Role: LEADER
I20260926 20:47:38.410287 20745 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:38.410416 20750 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1268dcbc3244b08820a093d7b87ae6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40427 } }
I20260926 20:47:38.410738 20750 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1268dcbc3244b08820a093d7b87ae6d. Latest consensus state: current_term: 1 leader_uuid: "b1268dcbc3244b08820a093d7b87ae6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1268dcbc3244b08820a093d7b87ae6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40427 } } }
I20260926 20:47:38.410944 20750 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:38.410876 20752 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1268dcbc3244b08820a093d7b87ae6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1268dcbc3244b08820a093d7b87ae6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40427 } } }
I20260926 20:47:38.411357 20752 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:38.411633 20759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:38.412189 20759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:38.413918 20759 catalog_manager.cc:1347] Generated new cluster ID: 51254d3a11884c0492d9e241d84adbcf
I20260926 20:47:38.413970 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:38.407059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34651 (local address 127.0.0.1:40427)
0926 20:47:38.407267 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:38.407277 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:38.407310 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:47:38.408142 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:38.408151 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:38.408215 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:38.408363 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:38.408542 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:38.410197 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:38.411451 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:38.412083 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:38.412191 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:38.413296 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:38.413300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:38.413300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:38.413470 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:38.413506 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:38.413517 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:38.413616 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:38.413720 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 20:47:38.413748 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260926 20:47:38.414031 20759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:47:38.415067 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:47:38.422824 20759 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:38.423591 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:38.420048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49913 (local address 127.0.0.1:40427)
0926 20:47:38.420086 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:38.420088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:38.420276 (+ 188us) server_negotiation.cc:407] Connection header received
0926 20:47:38.420334 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:38.420334 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:38.420344 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:38.420395 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:38.420548 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:38.421250 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:38.422308 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:38.422590 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:38.422668 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:38.423318 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:38.423320 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:38.423320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:38.423324 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:38.423339 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:38.423344 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:38.423406 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:38.423510 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 20:47:38.423520 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:47:38.423854 20759 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:38.430279 20759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1268dcbc3244b08820a093d7b87ae6d: Generated new TSK 0
I20260926 20:47:38.430447 20759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:38.436162 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 20677
W20260926 20:47:38.440227 20662 connection.cc:535] client connection to 127.0.0.1:40427 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40427): Connection reset by peer (error 104)
2026-09-26T20:47:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:38 dist-test-slave-4hd3 krb5kdc[20776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:38 dist-test-slave-4hd3 krb5kdc[20776](info): set up 2 sockets
Sep 26 20:47:38 dist-test-slave-4hd3 krb5kdc[20776](info): commencing operation
krb5kdc: starting...
W20260926 20:47:39.467893 393 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:39 dist-test-slave-4hd3 krb5kdc[20776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:47:39.584344 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35829
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:39.667048 20789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:39.667260 20789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:39.667284 20789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:39.668556 20789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:39.668603 20789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:39.668617 20789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:39.668630 20789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:39.668666 20789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:39.670153 20789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35829
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:39.670399 20789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:39.670644 20789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:39.675530 20795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:39.675544 20797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:39.675685 20789 server_base.cc:1034] running on GCE node
W20260926 20:47:39.675544 20794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:39.676108 20789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:39.676425 20789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:39.677624 20789 hybrid_clock.cc:648] HybridClock initialized: now 1790455659677596 us; error 38 us; skew 500 ppm
Sep 26 20:47:39 dist-test-slave-4hd3 krb5kdc[20776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:39.680250 20789 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:47:39.680732 20789 webserver.cc:460] Webserver started at http://127.0.0.1:41985/ using document root <none> and password file <none>
I20260926 20:47:39.680960 20789 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:39.681037 20789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:39.681186 20789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:39.682264 20789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "f7ac5710628e4b51876b8fb43292e696"
format_stamp: "Formatted at 2026-09-26 20:47:39 on dist-test-slave-4hd3"
server_key: "73217c0bcff9ed389e0b440491a0df60"
server_key_iv: "49bafa4e75c9943153df6bd4025bc90e"
server_key_version: "encryptionkey@0"
I20260926 20:47:39.682628 20789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "f7ac5710628e4b51876b8fb43292e696"
format_stamp: "Formatted at 2026-09-26 20:47:39 on dist-test-slave-4hd3"
server_key: "73217c0bcff9ed389e0b440491a0df60"
server_key_iv: "49bafa4e75c9943153df6bd4025bc90e"
server_key_version: "encryptionkey@0"
I20260926 20:47:39.684001 20789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:47:39.684725 20804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:39.684898 20789 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:47:39.685010 20789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "f7ac5710628e4b51876b8fb43292e696"
format_stamp: "Formatted at 2026-09-26 20:47:39 on dist-test-slave-4hd3"
server_key: "73217c0bcff9ed389e0b440491a0df60"
server_key_iv: "49bafa4e75c9943153df6bd4025bc90e"
server_key_version: "encryptionkey@0"
I20260926 20:47:39.685083 20789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:39.688263 20789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:39.690069 20789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:39.690213 20789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:39.693910 20789 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35829
I20260926 20:47:39.693945 20856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35829 every 8 connection(s)
I20260926 20:47:39.694267 20789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:39.695247 20857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:39.697546 20857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696: Bootstrap starting.
I20260926 20:47:39.698098 20857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:39.698377 20857 log.cc:824] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:39.698854 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 20789
I20260926 20:47:39.698982 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:39.699163 393 external_mini_cluster.cc:1428] Setting key 590b5621e5d3c712b4216e2ebb8af54a
I20260926 20:47:39.699160 20857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696: No bootstrap required, opened a new log
I20260926 20:47:39.701493 20857 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ac5710628e4b51876b8fb43292e696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35829 } }
I20260926 20:47:39.701665 20857 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:39.701706 20857 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ac5710628e4b51876b8fb43292e696, State: Initialized, Role: FOLLOWER
I20260926 20:47:39.701830 20857 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ac5710628e4b51876b8fb43292e696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35829 } }
I20260926 20:47:39.701941 20857 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:39.702010 20857 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:39.702059 20857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:39.702325 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:39.699739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50507 (local address 127.0.0.1:35829)
0926 20:47:39.699930 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:39.699941 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:39.699975 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:47:39.700632 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:39.700640 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:39.700675 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:39.700820 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:39.701523 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:39.701527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:39.701527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:39.701532 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:39.701572 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:39.701586 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:39.701662 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:39.701861 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 20:47:39.701923 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260926 20:47:39.702731 20857 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ac5710628e4b51876b8fb43292e696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35829 } }
I20260926 20:47:39.702834 20857 leader_election.cc:302] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7ac5710628e4b51876b8fb43292e696; no voters:
I20260926 20:47:39.703038 20857 leader_election.cc:288] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:39.703064 20862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:39.703214 20862 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696 [term 1 LEADER]: Becoming Leader. State: Replica: f7ac5710628e4b51876b8fb43292e696, State: Running, Role: LEADER
I20260926 20:47:39.703287 20862 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ac5710628e4b51876b8fb43292e696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35829 } }
I20260926 20:47:39.703313 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:47:39.703331 20857 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:39.704664 20863 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7ac5710628e4b51876b8fb43292e696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ac5710628e4b51876b8fb43292e696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35829 } } }
I20260926 20:47:39.704768 20863 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:39.704918 20864 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7ac5710628e4b51876b8fb43292e696. Latest consensus state: current_term: 1 leader_uuid: "f7ac5710628e4b51876b8fb43292e696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ac5710628e4b51876b8fb43292e696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35829 } } }
I20260926 20:47:39.705008 20864 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696 [sys.catalog]: This master's current role is: LEADER
W20260926 20:47:39.705529 20881 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:47:39.705593 20881 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:47:39.705713 20882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:39.705889 20882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:39.706192 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:39.704849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53985 (local address 127.0.0.1:35829)
0926 20:47:39.704881 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:39.704882 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:39.705097 (+ 215us) server_negotiation.cc:407] Connection header received
0926 20:47:39.705189 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:39.705190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:39.705194 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:39.705222 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:39.705927 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:39.705928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:39.705929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:39.705933 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:39.705945 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:39.705949 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:39.705980 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:39.706128 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 20:47:39.706141 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 20:47:39.707454 20882 catalog_manager.cc:1347] Generated new cluster ID: d89237213c494cc8be03023f438096d4
I20260926 20:47:39.707504 20882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:47:39.721766 20882 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:39.721946 20882 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:39.731990 20882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7ac5710628e4b51876b8fb43292e696: Generated new TSK 0
I20260926 20:47:39.732203 20882 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:39.738688 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 20789
2026-09-26T20:47:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:39 dist-test-slave-4hd3 krb5kdc[20888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:39 dist-test-slave-4hd3 krb5kdc[20888](info): set up 2 sockets
Sep 26 20:47:39 dist-test-slave-4hd3 krb5kdc[20888](info): commencing operation
krb5kdc: starting...
W20260926 20:47:40.763638 393 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:40 dist-test-slave-4hd3 krb5kdc[20888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:47:40.880007 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40191
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:40.963335 20901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:40.963547 20901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:40.963570 20901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:40.964839 20901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:40.964890 20901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:40.964903 20901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:40.964916 20901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:40.964928 20901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:40.966509 20901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:40.966764 20901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:40.967016 20901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:40.971966 20907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:40.971985 20906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:40.972002 20909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:40.972272 20901 server_base.cc:1034] running on GCE node
I20260926 20:47:40.972535 20901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:40.972822 20901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:40.974010 20901 hybrid_clock.cc:648] HybridClock initialized: now 1790455660973988 us; error 38 us; skew 500 ppm
Sep 26 20:47:40 dist-test-slave-4hd3 krb5kdc[20888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:40.976656 20901 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:47:40.977175 20901 webserver.cc:460] Webserver started at http://127.0.0.1:43081/ using document root <none> and password file <none>
I20260926 20:47:40.977406 20901 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:40.977475 20901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:40.977638 20901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:40.978771 20901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "dd663a7512314bf9a09fc9b1bccea9b2"
format_stamp: "Formatted at 2026-09-26 20:47:40 on dist-test-slave-4hd3"
server_key: "b8cda937a8a1768f7c249ab74f907e54"
server_key_iv: "2e93bd81f385be04c5121edb54c641cb"
server_key_version: "encryptionkey@0"
I20260926 20:47:40.979153 20901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "dd663a7512314bf9a09fc9b1bccea9b2"
format_stamp: "Formatted at 2026-09-26 20:47:40 on dist-test-slave-4hd3"
server_key: "b8cda937a8a1768f7c249ab74f907e54"
server_key_iv: "2e93bd81f385be04c5121edb54c641cb"
server_key_version: "encryptionkey@0"
I20260926 20:47:40.980620 20901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:47:40.981345 20916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:40.981508 20901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:47:40.981630 20901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "dd663a7512314bf9a09fc9b1bccea9b2"
format_stamp: "Formatted at 2026-09-26 20:47:40 on dist-test-slave-4hd3"
server_key: "b8cda937a8a1768f7c249ab74f907e54"
server_key_iv: "2e93bd81f385be04c5121edb54c641cb"
server_key_version: "encryptionkey@0"
I20260926 20:47:40.981707 20901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:41.028754 20901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:41.031176 20901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:41.031340 20901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:41.035477 20901 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40191
I20260926 20:47:41.035530 20968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40191 every 8 connection(s)
I20260926 20:47:41.035923 20901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:41.036753 20969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:41.038944 20969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2: Bootstrap starting.
I20260926 20:47:41.039455 20969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:41.039721 20969 log.cc:824] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:41.040422 20969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2: No bootstrap required, opened a new log
I20260926 20:47:41.041875 20969 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd663a7512314bf9a09fc9b1bccea9b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40191 } }
I20260926 20:47:41.042007 20969 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:41.042029 20969 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd663a7512314bf9a09fc9b1bccea9b2, State: Initialized, Role: FOLLOWER
I20260926 20:47:41.042182 20969 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd663a7512314bf9a09fc9b1bccea9b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40191 } }
I20260926 20:47:41.042266 20969 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:41.042289 20969 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:41.042320 20969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:41.042776 20969 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd663a7512314bf9a09fc9b1bccea9b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40191 } }
I20260926 20:47:41.042856 20969 leader_election.cc:302] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd663a7512314bf9a09fc9b1bccea9b2; no voters:
I20260926 20:47:41.042973 20969 leader_election.cc:288] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:41.043025 20972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:41.043222 20972 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2 [term 1 LEADER]: Becoming Leader. State: Replica: dd663a7512314bf9a09fc9b1bccea9b2, State: Running, Role: LEADER
I20260926 20:47:41.043234 20969 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:41.043383 20972 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd663a7512314bf9a09fc9b1bccea9b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40191 } }
I20260926 20:47:41.043830 20973 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd663a7512314bf9a09fc9b1bccea9b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd663a7512314bf9a09fc9b1bccea9b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40191 } } }
I20260926 20:47:41.043908 20973 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:41.043962 20974 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd663a7512314bf9a09fc9b1bccea9b2. Latest consensus state: current_term: 1 leader_uuid: "dd663a7512314bf9a09fc9b1bccea9b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd663a7512314bf9a09fc9b1bccea9b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40191 } } }
I20260926 20:47:41.044047 20974 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:41.044337 20980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:41.044878 20980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:41.044978 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 20901
I20260926 20:47:41.045099 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:41.045288 393 external_mini_cluster.cc:1428] Setting key 92e7831d828b5ca5560eb09d65ba547e
I20260926 20:47:41.046533 20980 catalog_manager.cc:1347] Generated new cluster ID: 1c96200a04484f5c83c9e264d8485585
I20260926 20:47:41.046586 20980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:47:41 dist-test-slave-4hd3 krb5kdc[20888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:47:41.057195 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:41.045723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52095 (local address 127.0.0.1:40191)
0926 20:47:41.046255 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:41.046264 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:41.046295 (+ 31us) server_negotiation.cc:407] Connection header received
0926 20:47:41.046409 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:41.046416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:41.046477 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:41.046658 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:41.047352 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:41.048849 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:41.049853 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:41.050479 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:41.050585 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:41.053383 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:41.053387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:41.053387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:41.053399 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:41.055308 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:41.055754 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:41.055757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:41.055757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:41.055878 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:41.056094 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:41.056096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:41.056096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:41.056408 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:41.056594 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:41.056663 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 20:47:41.056922 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":69,"threads_started":1}
I20260926 20:47:41.057504 20980 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:41.058195 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:47:41.058212 20980 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:41.064265 20980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd663a7512314bf9a09fc9b1bccea9b2: Generated new TSK 0
I20260926 20:47:41.064463 20980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:41.066784 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:41.059744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41605 (local address 127.0.0.1:40191)
0926 20:47:41.059782 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:41.059783 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:41.059925 (+ 142us) server_negotiation.cc:407] Connection header received
0926 20:47:41.059985 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:41.059986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:41.059997 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:41.060050 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:41.060676 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:41.061403 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:41.062478 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:41.062787 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:41.062844 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:41.064247 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:41.064248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:41.064249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:41.064258 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:41.065768 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:41.066152 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:41.066153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:41.066153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:41.066200 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:41.066390 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:41.066391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:41.066391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:41.066477 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:41.066547 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:41.066673 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 20:47:41.066730 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:47:41.069216 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 20901
2026-09-26T20:47:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:41 dist-test-slave-4hd3 krb5kdc[21000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:41 dist-test-slave-4hd3 krb5kdc[21000](info): set up 2 sockets
Sep 26 20:47:41 dist-test-slave-4hd3 krb5kdc[21000](info): commencing operation
krb5kdc: starting...
W20260926 20:47:42.104131 393 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:42 dist-test-slave-4hd3 krb5kdc[21000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:47:42.216741 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45515
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:42.297173 21013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:42.297369 21013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:42.297391 21013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:42.298486 21013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:42.298522 21013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:42.298537 21013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:42.298550 21013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:42.298563 21013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:42.299954 21013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:42.300145 21013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:42.300303 21013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:42.305114 21018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:42.305169 21013 server_base.cc:1034] running on GCE node
W20260926 20:47:42.305132 21021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:42.305154 21019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:42.305617 21013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:42.305945 21013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:42.307139 21013 hybrid_clock.cc:648] HybridClock initialized: now 1790455662307131 us; error 39 us; skew 500 ppm
Sep 26 20:47:42 dist-test-slave-4hd3 krb5kdc[21000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:42.309644 21013 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:47:42.310077 21013 webserver.cc:460] Webserver started at http://127.0.0.1:45143/ using document root <none> and password file <none>
I20260926 20:47:42.310295 21013 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:42.310361 21013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:42.310515 21013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:42.311546 21013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "fc3a2d2cfb744e698dc60255c41ee744"
format_stamp: "Formatted at 2026-09-26 20:47:42 on dist-test-slave-4hd3"
server_key: "4c45ce7dd6b78c8bcce0b293fb3cc71f"
server_key_iv: "fec89f49d33870512953e7f0cf8cf9c9"
server_key_version: "encryptionkey@0"
I20260926 20:47:42.311935 21013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "fc3a2d2cfb744e698dc60255c41ee744"
format_stamp: "Formatted at 2026-09-26 20:47:42 on dist-test-slave-4hd3"
server_key: "4c45ce7dd6b78c8bcce0b293fb3cc71f"
server_key_iv: "fec89f49d33870512953e7f0cf8cf9c9"
server_key_version: "encryptionkey@0"
I20260926 20:47:42.313241 21013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:47:42.314036 21028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:42.314206 21013 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:47:42.314312 21013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "fc3a2d2cfb744e698dc60255c41ee744"
format_stamp: "Formatted at 2026-09-26 20:47:42 on dist-test-slave-4hd3"
server_key: "4c45ce7dd6b78c8bcce0b293fb3cc71f"
server_key_iv: "fec89f49d33870512953e7f0cf8cf9c9"
server_key_version: "encryptionkey@0"
I20260926 20:47:42.314393 21013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:42.334262 21013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:42.336167 21013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:42.336335 21013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:42.340219 21013 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45515
I20260926 20:47:42.340292 21080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45515 every 8 connection(s)
I20260926 20:47:42.340680 21013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:42.341156 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 21013
I20260926 20:47:42.341295 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:42.341512 393 external_mini_cluster.cc:1428] Setting key 666fe457fc9da6a1e6ca98b9d116ed35
I20260926 20:47:42.341820 21081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:42.344962 21081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744: Bootstrap starting.
I20260926 20:47:42.345577 21081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:42.345871 21081 log.cc:824] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:42.346550 21081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744: No bootstrap required, opened a new log
I20260926 20:47:42.347324 21084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:42.342069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52403 (local address 127.0.0.1:45515)
0926 20:47:42.342265 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:42.342275 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:42.342309 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:47:42.342946 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:42.342954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:42.343036 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:42.343222 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:42.343436 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:42.344427 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:42.345308 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:42.345761 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:42.345887 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:42.346610 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:42.346615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:42.346615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:42.346812 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:42.346843 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:42.346855 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:42.346927 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:42.346998 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 20:47:42.347165 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260926 20:47:42.348361 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:47:42.348605 21081 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3a2d2cfb744e698dc60255c41ee744" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45515 } }
I20260926 20:47:42.348769 21081 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:42.348809 21081 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc3a2d2cfb744e698dc60255c41ee744, State: Initialized, Role: FOLLOWER
I20260926 20:47:42.348935 21081 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3a2d2cfb744e698dc60255c41ee744" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45515 } }
I20260926 20:47:42.349009 21081 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:42.349041 21081 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:42.349092 21081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:42.349871 21081 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3a2d2cfb744e698dc60255c41ee744" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45515 } }
I20260926 20:47:42.350014 21081 leader_election.cc:302] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc3a2d2cfb744e698dc60255c41ee744; no voters:
I20260926 20:47:42.350199 21081 leader_election.cc:288] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:42.350229 21091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:42.350449 21081 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:42.350457 21091 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744 [term 1 LEADER]: Becoming Leader. State: Replica: fc3a2d2cfb744e698dc60255c41ee744, State: Running, Role: LEADER
I20260926 20:47:42.350590 21091 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3a2d2cfb744e698dc60255c41ee744" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45515 } }
I20260926 20:47:42.350976 21091 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc3a2d2cfb744e698dc60255c41ee744. Latest consensus state: current_term: 1 leader_uuid: "fc3a2d2cfb744e698dc60255c41ee744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3a2d2cfb744e698dc60255c41ee744" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45515 } } }
I20260926 20:47:42.351053 21091 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:42.351279 21100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:42.351567 21092 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc3a2d2cfb744e698dc60255c41ee744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3a2d2cfb744e698dc60255c41ee744" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45515 } } }
I20260926 20:47:42.351738 21092 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:42.352010 21100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:42.353737 21084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:42.349898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53263 (local address 127.0.0.1:45515)
0926 20:47:42.349956 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:42.349958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:42.350082 (+ 124us) server_negotiation.cc:407] Connection header received
0926 20:47:42.350181 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:42.350183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:42.350195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:42.350268 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:42.350505 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:42.351254 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:42.352318 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:42.352632 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:42.352708 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:42.353375 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:42.353376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:42.353376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:42.353381 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:42.353396 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:42.353400 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:42.353449 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:42.353640 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 20:47:42.353665 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:47:42.353899 21100 catalog_manager.cc:1347] Generated new cluster ID: 3a6dd05ff2264ad3884ba6b2b9d584e2
I20260926 20:47:42.353945 21100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:47:42.370353 21100 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:42.371083 21100 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:42.377035 21100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc3a2d2cfb744e698dc60255c41ee744: Generated new TSK 0
I20260926 20:47:42.377218 21100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:42.381317 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 21013
2026-09-26T20:47:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:42 dist-test-slave-4hd3 krb5kdc[21112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:42 dist-test-slave-4hd3 krb5kdc[21112](info): set up 2 sockets
Sep 26 20:47:42 dist-test-slave-4hd3 krb5kdc[21112](info): commencing operation
krb5kdc: starting...
W20260926 20:47:43.411761 393 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:43 dist-test-slave-4hd3 krb5kdc[21112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:47:43.524478 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41569
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:43.604998 21125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:43.605196 21125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:43.605218 21125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:43.606307 21125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:43.606341 21125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:43.606355 21125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:43.606369 21125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:43.606380 21125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:43.607928 21125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:43.608124 21125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:43.608287 21125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:43.613029 21133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:43.613034 21130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:43.613142 21131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:43.613302 21125 server_base.cc:1034] running on GCE node
I20260926 20:47:43.613658 21125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:43.613896 21125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:43.615051 21125 hybrid_clock.cc:648] HybridClock initialized: now 1790455663615038 us; error 28 us; skew 500 ppm
Sep 26 20:47:43 dist-test-slave-4hd3 krb5kdc[21112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:43.622269 21125 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:47:43.622740 21125 webserver.cc:460] Webserver started at http://127.0.0.1:45855/ using document root <none> and password file <none>
I20260926 20:47:43.622978 21125 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:43.623025 21125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:43.623195 21125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:43.624356 21125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "a1b82c1fc9dc4fc8b00a9cfcdfe1116c"
format_stamp: "Formatted at 2026-09-26 20:47:43 on dist-test-slave-4hd3"
server_key: "ae0027b5d67940bc7952e2d3726dca22"
server_key_iv: "5365cb4fcb974f3820c8e27b5ae5605b"
server_key_version: "encryptionkey@0"
I20260926 20:47:43.624722 21125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "a1b82c1fc9dc4fc8b00a9cfcdfe1116c"
format_stamp: "Formatted at 2026-09-26 20:47:43 on dist-test-slave-4hd3"
server_key: "ae0027b5d67940bc7952e2d3726dca22"
server_key_iv: "5365cb4fcb974f3820c8e27b5ae5605b"
server_key_version: "encryptionkey@0"
I20260926 20:47:43.626070 21125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:47:43.626878 21140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:43.627141 21125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:43.627246 21125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "a1b82c1fc9dc4fc8b00a9cfcdfe1116c"
format_stamp: "Formatted at 2026-09-26 20:47:43 on dist-test-slave-4hd3"
server_key: "ae0027b5d67940bc7952e2d3726dca22"
server_key_iv: "5365cb4fcb974f3820c8e27b5ae5605b"
server_key_version: "encryptionkey@0"
I20260926 20:47:43.627324 21125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:43.630769 21125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:43.632964 21125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:43.633107 21125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:43.636976 21125 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41569
I20260926 20:47:43.637030 21192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41569 every 8 connection(s)
I20260926 20:47:43.637375 21125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:43.638305 21193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:43.638592 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 21125
I20260926 20:47:43.638711 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:43.638895 393 external_mini_cluster.cc:1428] Setting key 842a0d9ffc536a965378c8f95847e008
I20260926 20:47:43.641005 21193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c: Bootstrap starting.
I20260926 20:47:43.641364 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:43.639443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34623 (local address 127.0.0.1:41569)
0926 20:47:43.639639 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:43.639648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:43.639685 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:47:43.640140 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:43.640146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:43.640171 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:43.640321 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:43.640952 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:43.640956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:43.640956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:43.640960 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:43.641003 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:43.641019 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:43.641095 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:43.641164 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 20:47:43.641206 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260926 20:47:43.641764 21193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:43.642099 21193 log.cc:824] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:43.642311 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:47:43.642793 21193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c: No bootstrap required, opened a new log
I20260926 20:47:43.644748 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:43.643765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51759 (local address 127.0.0.1:41569)
0926 20:47:43.643809 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:43.643811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:43.643837 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:47:43.643907 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:43.643908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:43.643913 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:43.643966 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:43.644542 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:43.644544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:43.644544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:43.644548 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:43.644561 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:43.644565 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:43.644620 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:43.644672 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 20:47:43.644686 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:47:43.644950 21193 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b82c1fc9dc4fc8b00a9cfcdfe1116c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41569 } }
I20260926 20:47:43.645107 21193 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:43.645165 21193 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b82c1fc9dc4fc8b00a9cfcdfe1116c, State: Initialized, Role: FOLLOWER
I20260926 20:47:43.645299 21193 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b82c1fc9dc4fc8b00a9cfcdfe1116c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41569 } }
I20260926 20:47:43.645370 21193 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:43.645391 21193 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:43.645418 21193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:43.645892 21193 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b82c1fc9dc4fc8b00a9cfcdfe1116c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41569 } }
I20260926 20:47:43.645975 21193 leader_election.cc:302] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1b82c1fc9dc4fc8b00a9cfcdfe1116c; no voters:
I20260926 20:47:43.646103 21193 leader_election.cc:288] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:43.646147 21203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:43.646306 21203 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c [term 1 LEADER]: Becoming Leader. State: Replica: a1b82c1fc9dc4fc8b00a9cfcdfe1116c, State: Running, Role: LEADER
I20260926 20:47:43.646374 21193 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:43.646526 21203 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b82c1fc9dc4fc8b00a9cfcdfe1116c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41569 } }
I20260926 20:47:43.646886 21204 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1b82c1fc9dc4fc8b00a9cfcdfe1116c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b82c1fc9dc4fc8b00a9cfcdfe1116c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41569 } } }
I20260926 20:47:43.647001 21204 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:43.647207 21205 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1b82c1fc9dc4fc8b00a9cfcdfe1116c. Latest consensus state: current_term: 1 leader_uuid: "a1b82c1fc9dc4fc8b00a9cfcdfe1116c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b82c1fc9dc4fc8b00a9cfcdfe1116c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41569 } } }
I20260926 20:47:43.647280 21205 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:43.647496 21211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:43.647979 21211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:43.649514 21211 catalog_manager.cc:1347] Generated new cluster ID: 198ede42ec8e4e3e9cab8cfd2dee0785
I20260926 20:47:43.649566 21211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:47:43.664256 21211 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:43.664438 21211 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:43.668212 21211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1b82c1fc9dc4fc8b00a9cfcdfe1116c: Generated new TSK 0
I20260926 20:47:43.668401 21211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:43.674618 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 21125
2026-09-26T20:47:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:43 dist-test-slave-4hd3 krb5kdc[21224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:43 dist-test-slave-4hd3 krb5kdc[21224](info): set up 2 sockets
Sep 26 20:47:43 dist-test-slave-4hd3 krb5kdc[21224](info): commencing operation
krb5kdc: starting...
W20260926 20:47:44.695935 393 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:44 dist-test-slave-4hd3 krb5kdc[21224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:47:44.810398 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38491
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:44.889961 21237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:44.890209 21237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:44.890245 21237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:44.891422 21237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:44.891497 21237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:44.891522 21237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:44.891542 21237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:44.891603 21237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:44.893215 21237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:44.893488 21237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:44.893718 21237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:44.898725 21243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:44.898671 21245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:44.898670 21242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:44.898888 21237 server_base.cc:1034] running on GCE node
I20260926 20:47:44.899170 21237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:44.899452 21237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:44.900595 21237 hybrid_clock.cc:648] HybridClock initialized: now 1790455664900586 us; error 32 us; skew 500 ppm
Sep 26 20:47:44 dist-test-slave-4hd3 krb5kdc[21224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:44.903143 21237 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:47:44.903630 21237 webserver.cc:460] Webserver started at http://127.0.0.1:42171/ using document root <none> and password file <none>
I20260926 20:47:44.903895 21237 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:44.903964 21237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:44.904140 21237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:44.905755 21237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "087fe00e8d0d4010bbd08f1790a59eab"
format_stamp: "Formatted at 2026-09-26 20:47:44 on dist-test-slave-4hd3"
server_key: "0b09cc613f4cbc2ee5eebcccee1715e6"
server_key_iv: "0ce81b9e468b7fa4e9ee0d6d18545053"
server_key_version: "encryptionkey@0"
I20260926 20:47:44.906175 21237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "087fe00e8d0d4010bbd08f1790a59eab"
format_stamp: "Formatted at 2026-09-26 20:47:44 on dist-test-slave-4hd3"
server_key: "0b09cc613f4cbc2ee5eebcccee1715e6"
server_key_iv: "0ce81b9e468b7fa4e9ee0d6d18545053"
server_key_version: "encryptionkey@0"
I20260926 20:47:44.908020 21237 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 20:47:44.909045 21252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:44.909227 21237 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:44.909328 21237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "087fe00e8d0d4010bbd08f1790a59eab"
format_stamp: "Formatted at 2026-09-26 20:47:44 on dist-test-slave-4hd3"
server_key: "0b09cc613f4cbc2ee5eebcccee1715e6"
server_key_iv: "0ce81b9e468b7fa4e9ee0d6d18545053"
server_key_version: "encryptionkey@0"
I20260926 20:47:44.909413 21237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:44.927197 21237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:44.929091 21237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:44.929251 21237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:44.933249 21237 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38491
I20260926 20:47:44.933281 21304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38491 every 8 connection(s)
I20260926 20:47:44.933629 21237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:44.934463 21305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:44.934680 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 21237
I20260926 20:47:44.934798 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:44.935009 393 external_mini_cluster.cc:1428] Setting key 2123e64b15669604cfc496e6c43d3fcc
I20260926 20:47:44.937981 21305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab: Bootstrap starting.
I20260926 20:47:44.938706 21305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:44.939004 21305 log.cc:824] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:44.939689 21305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab: No bootstrap required, opened a new log
Sep 26 20:47:44 dist-test-slave-4hd3 krb5kdc[21224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:47:44.941968 21305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087fe00e8d0d4010bbd08f1790a59eab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38491 } }
I20260926 20:47:44.942123 21305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:44.942152 21305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 087fe00e8d0d4010bbd08f1790a59eab, State: Initialized, Role: FOLLOWER
I20260926 20:47:44.942309 21305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087fe00e8d0d4010bbd08f1790a59eab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38491 } }
I20260926 20:47:44.942370 21305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:44.942392 21305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:44.942458 21305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:44.943099 21305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087fe00e8d0d4010bbd08f1790a59eab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38491 } }
I20260926 20:47:44.943218 21305 leader_election.cc:302] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 087fe00e8d0d4010bbd08f1790a59eab; no voters:
I20260926 20:47:44.943418 21305 leader_election.cc:288] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:44.943468 21310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:44.943692 21305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:44.944028 21310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab [term 1 LEADER]: Becoming Leader. State: Replica: 087fe00e8d0d4010bbd08f1790a59eab, State: Running, Role: LEADER
I20260926 20:47:44.944175 21310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087fe00e8d0d4010bbd08f1790a59eab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38491 } }
I20260926 20:47:44.944756 21311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "087fe00e8d0d4010bbd08f1790a59eab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087fe00e8d0d4010bbd08f1790a59eab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38491 } } }
I20260926 20:47:44.944837 21311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:44.945215 21312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 087fe00e8d0d4010bbd08f1790a59eab. Latest consensus state: current_term: 1 leader_uuid: "087fe00e8d0d4010bbd08f1790a59eab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087fe00e8d0d4010bbd08f1790a59eab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38491 } } }
I20260926 20:47:44.945300 21312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab [sys.catalog]: This master's current role is: LEADER
W20260926 20:47:44.945844 21327 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:47:44.945945 21327 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:47:44.946024 21320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:44.946214 21320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:44.946832 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:44.935559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:45873 (local address 127.0.0.1:38491)
0926 20:47:44.935754 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:44.935763 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:44.935796 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:47:44.936227 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:44.936234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:44.936281 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:44.936397 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:44.937190 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:44.938120 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:44.939216 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:44.939621 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:44.939773 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:44.942620 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:44.942623 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:44.942624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:44.942634 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:44.944888 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:44.945517 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:44.945520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:44.945520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:44.945641 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:44.945935 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:44.945936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:44.945936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:44.946215 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:44.946399 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:44.946527 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 20:47:44.946640 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260926 20:47:44.947741 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:47:44.947821 21320 catalog_manager.cc:1347] Generated new cluster ID: bf4802ef06df4b57b775f80777fc13df
I20260926 20:47:44.947865 21320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:47:44.956429 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:44.949159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:48489 (local address 127.0.0.1:38491)
0926 20:47:44.949193 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:44.949195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:44.949289 (+ 94us) server_negotiation.cc:407] Connection header received
0926 20:47:44.949334 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:44.949335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:44.949344 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:44.949409 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:44.949933 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:44.950679 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:44.951397 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:44.951743 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:44.951802 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:44.953450 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:44.953452 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:44.953452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:44.953459 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:44.955064 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:44.955503 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:44.955504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:44.955504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:44.955559 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:44.955881 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:44.955882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:44.955882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:44.956005 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:44.956104 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:44.956277 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 20:47:44.956339 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:47:44.957324 21320 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:44.957794 21320 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:44.965339 21320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 087fe00e8d0d4010bbd08f1790a59eab: Generated new TSK 0
I20260926 20:47:44.965564 21320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:44.974009 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 21237
W20260926 20:47:44.978602 21222 connection.cc:535] client connection to 127.0.0.1:38491 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38491): Connection reset by peer (error 104)
2026-09-26T20:47:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:45 dist-test-slave-4hd3 krb5kdc[21337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:45 dist-test-slave-4hd3 krb5kdc[21337](info): set up 2 sockets
Sep 26 20:47:45 dist-test-slave-4hd3 krb5kdc[21337](info): commencing operation
krb5kdc: starting...
W20260926 20:47:46.003855 393 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:46 dist-test-slave-4hd3 krb5kdc[21337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:47:46.120565 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:46.201550 21350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:46.201762 21350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:46.201785 21350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:46.202849 21350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:46.202883 21350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:46.202898 21350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:46.202909 21350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:46.202921 21350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:46.204311 21350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:46.204487 21350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:46.204648 21350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:46.209343 21356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:46.209339 21358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:46.209339 21355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:46.209728 21350 server_base.cc:1034] running on GCE node
I20260926 20:47:46.209981 21350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:46.210210 21350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:46.211344 21350 hybrid_clock.cc:648] HybridClock initialized: now 1790455666211334 us; error 33 us; skew 500 ppm
Sep 26 20:47:46 dist-test-slave-4hd3 krb5kdc[21337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:46.213742 21350 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:47:46.214155 21350 webserver.cc:460] Webserver started at http://127.0.0.1:41917/ using document root <none> and password file <none>
I20260926 20:47:46.214380 21350 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:46.214449 21350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:46.214603 21350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:46.215660 21350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "62a9fc662c444b14bf817e8c0f211c78"
format_stamp: "Formatted at 2026-09-26 20:47:46 on dist-test-slave-4hd3"
server_key: "d9b2198745dd6b40ef4af2256ea2e62b"
server_key_iv: "f546361d62d5f77708f89ba10ca19fe2"
server_key_version: "encryptionkey@0"
I20260926 20:47:46.216087 21350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "62a9fc662c444b14bf817e8c0f211c78"
format_stamp: "Formatted at 2026-09-26 20:47:46 on dist-test-slave-4hd3"
server_key: "d9b2198745dd6b40ef4af2256ea2e62b"
server_key_iv: "f546361d62d5f77708f89ba10ca19fe2"
server_key_version: "encryptionkey@0"
I20260926 20:47:46.217386 21350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:46.218173 21365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:46.218405 21350 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:46.218526 21350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "62a9fc662c444b14bf817e8c0f211c78"
format_stamp: "Formatted at 2026-09-26 20:47:46 on dist-test-slave-4hd3"
server_key: "d9b2198745dd6b40ef4af2256ea2e62b"
server_key_iv: "f546361d62d5f77708f89ba10ca19fe2"
server_key_version: "encryptionkey@0"
I20260926 20:47:46.218623 21350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:46.233774 21350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:46.236052 21350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:46.236234 21350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:46.240209 21350 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45009
I20260926 20:47:46.240249 21417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45009 every 8 connection(s)
I20260926 20:47:46.240607 21350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:46.241488 21418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:46.243760 21418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78: Bootstrap starting.
I20260926 20:47:46.244385 21418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:46.244678 21418 log.cc:824] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:46.245349 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 21350
I20260926 20:47:46.245456 21418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78: No bootstrap required, opened a new log
I20260926 20:47:46.245442 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:46.245689 393 external_mini_cluster.cc:1428] Setting key f39833ad6ff7416ac560d80f4488cc01
I20260926 20:47:46.248456 21418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a9fc662c444b14bf817e8c0f211c78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45009 } }
I20260926 20:47:46.248626 21418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:46.248656 21418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62a9fc662c444b14bf817e8c0f211c78, State: Initialized, Role: FOLLOWER
I20260926 20:47:46.248792 21418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a9fc662c444b14bf817e8c0f211c78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45009 } }
I20260926 20:47:46.248884 21418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:46.248915 21418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:46.248992 21418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:46.249536 21418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a9fc662c444b14bf817e8c0f211c78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45009 } }
I20260926 20:47:46.249624 21418 leader_election.cc:302] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62a9fc662c444b14bf817e8c0f211c78; no voters:
I20260926 20:47:46.249805 21418 leader_election.cc:288] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:46.249881 21423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:46.250037 21418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:46.250056 21423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78 [term 1 LEADER]: Becoming Leader. State: Replica: 62a9fc662c444b14bf817e8c0f211c78, State: Running, Role: LEADER
I20260926 20:47:46.250141 21423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a9fc662c444b14bf817e8c0f211c78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45009 } }
I20260926 20:47:46.250531 21423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62a9fc662c444b14bf817e8c0f211c78. Latest consensus state: current_term: 1 leader_uuid: "62a9fc662c444b14bf817e8c0f211c78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a9fc662c444b14bf817e8c0f211c78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45009 } } }
I20260926 20:47:46.250677 21423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:46.250706 21424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62a9fc662c444b14bf817e8c0f211c78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a9fc662c444b14bf817e8c0f211c78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45009 } } }
I20260926 20:47:46.250814 21424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:46.250983 21432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:46.251534 21432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:46.251940 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:46.246253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:49071 (local address 127.0.0.1:45009)
0926 20:47:46.246438 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:46.246448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:46.246482 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:47:46.246922 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:46.246931 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:46.246995 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:46.247143 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:46.247310 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:46.248766 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:46.249681 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:46.250287 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:46.250397 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:46.251232 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:46.251235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:46.251236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:46.251441 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:46.251482 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:46.251496 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:46.251589 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:46.251704 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 20:47:46.251756 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260926 20:47:46.252919 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:47:46.253185 21432 catalog_manager.cc:1347] Generated new cluster ID: a1fd5daebb62408c8fbd04cae161b677
I20260926 20:47:46.253229 21432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:47:46.257359 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:46.254396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:33929 (local address 127.0.0.1:45009)
0926 20:47:46.254433 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:46.254435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:46.254569 (+ 134us) server_negotiation.cc:407] Connection header received
0926 20:47:46.254641 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:46.254642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:46.254650 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:46.254701 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:46.254855 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:46.255308 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:46.256143 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:46.256362 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:46.256445 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:46.257117 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:46.257118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:46.257119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:46.257123 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:46.257137 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:46.257141 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:46.257194 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:46.257261 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 20:47:46.257270 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:47:46.270963 21432 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:46.271598 21432 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:46.278667 21432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62a9fc662c444b14bf817e8c0f211c78: Generated new TSK 0
I20260926 20:47:46.278895 21432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:46.282759 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 21350
W20260926 20:47:46.287024 21335 connection.cc:535] client connection to 127.0.0.1:45009 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45009): Connection reset by peer (error 104)
2026-09-26T20:47:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:46 dist-test-slave-4hd3 krb5kdc[21449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:46 dist-test-slave-4hd3 krb5kdc[21449](info): set up 2 sockets
Sep 26 20:47:46 dist-test-slave-4hd3 krb5kdc[21449](info): commencing operation
krb5kdc: starting...
W20260926 20:47:47.343887 393 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:47 dist-test-slave-4hd3 krb5kdc[21449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:47:47.459208 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43495
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:47.538551 21462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:47.538755 21462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:47.538779 21462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:47.539945 21462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:47.539988 21462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:47.540002 21462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:47.540014 21462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:47.540026 21462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:47.541466 21462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:47.541666 21462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:47.541833 21462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:47.546778 21467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:47.546751 21468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:47.546751 21470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:47.547257 21462 server_base.cc:1034] running on GCE node
I20260926 20:47:47.547499 21462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:47.547820 21462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:47.548978 21462 hybrid_clock.cc:648] HybridClock initialized: now 1790455667548968 us; error 28 us; skew 500 ppm
Sep 26 20:47:47 dist-test-slave-4hd3 krb5kdc[21449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:47.551530 21462 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:47:47.552058 21462 webserver.cc:460] Webserver started at http://127.0.0.1:35151/ using document root <none> and password file <none>
I20260926 20:47:47.552312 21462 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:47.552359 21462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:47.552486 21462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:47.553622 21462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "a63996f9835d4a51a5248e71036c9a87"
format_stamp: "Formatted at 2026-09-26 20:47:47 on dist-test-slave-4hd3"
server_key: "3d0b4143a024233017573d093c9d1de8"
server_key_iv: "aa856c0920114f80fccae8cbcffbb7ae"
server_key_version: "encryptionkey@0"
I20260926 20:47:47.553982 21462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "a63996f9835d4a51a5248e71036c9a87"
format_stamp: "Formatted at 2026-09-26 20:47:47 on dist-test-slave-4hd3"
server_key: "3d0b4143a024233017573d093c9d1de8"
server_key_iv: "aa856c0920114f80fccae8cbcffbb7ae"
server_key_version: "encryptionkey@0"
I20260926 20:47:47.555320 21462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:47:47.556169 21477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:47.556450 21462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:47:47.556533 21462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "a63996f9835d4a51a5248e71036c9a87"
format_stamp: "Formatted at 2026-09-26 20:47:47 on dist-test-slave-4hd3"
server_key: "3d0b4143a024233017573d093c9d1de8"
server_key_iv: "aa856c0920114f80fccae8cbcffbb7ae"
server_key_version: "encryptionkey@0"
I20260926 20:47:47.556622 21462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:47.560082 21462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:47.562155 21462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:47.562328 21462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:47.566376 21462 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43495
I20260926 20:47:47.566449 21529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43495 every 8 connection(s)
I20260926 20:47:47.566756 21462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:47.567873 21530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:47.570139 21530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87: Bootstrap starting.
I20260926 20:47:47.570664 21530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:47.570894 21530 log.cc:824] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:47.571601 21530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87: No bootstrap required, opened a new log
I20260926 20:47:47.573231 21530 raft_consensus.cc:348] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63996f9835d4a51a5248e71036c9a87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43495 } }
I20260926 20:47:47.573370 21530 raft_consensus.cc:374] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:47.573391 21530 raft_consensus.cc:729] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a63996f9835d4a51a5248e71036c9a87, State: Initialized, Role: FOLLOWER
I20260926 20:47:47.573570 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 21462
I20260926 20:47:47.573561 21530 consensus_queue.cc:260] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63996f9835d4a51a5248e71036c9a87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43495 } }
I20260926 20:47:47.573639 21530 raft_consensus.cc:388] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:47.573694 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:47.573698 21530 raft_consensus.cc:482] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:47.573746 21530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:47.573894 393 external_mini_cluster.cc:1428] Setting key 17216b698a0e091a3d7d172316b737c2
I20260926 20:47:47.574568 21530 raft_consensus.cc:504] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63996f9835d4a51a5248e71036c9a87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43495 } }
I20260926 20:47:47.574730 21530 leader_election.cc:302] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a63996f9835d4a51a5248e71036c9a87; no voters:
I20260926 20:47:47.574896 21530 leader_election.cc:288] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:47.574988 21535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:47.575155 21530 sys_catalog.cc:564] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:47.575186 21535 raft_consensus.cc:686] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87 [term 1 LEADER]: Becoming Leader. State: Replica: a63996f9835d4a51a5248e71036c9a87, State: Running, Role: LEADER
I20260926 20:47:47.575421 21535 consensus_queue.cc:237] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63996f9835d4a51a5248e71036c9a87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43495 } }
I20260926 20:47:47.575781 21536 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a63996f9835d4a51a5248e71036c9a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63996f9835d4a51a5248e71036c9a87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43495 } } }
I20260926 20:47:47.575860 21536 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:47.576005 21537 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a63996f9835d4a51a5248e71036c9a87. Latest consensus state: current_term: 1 leader_uuid: "a63996f9835d4a51a5248e71036c9a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63996f9835d4a51a5248e71036c9a87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43495 } } }
I20260926 20:47:47.576099 21537 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:47.576463 21539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:47.576890 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:47.574441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:47049 (local address 127.0.0.1:43495)
0926 20:47:47.574709 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:47.574719 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:47.574756 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:47:47.574879 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:47.574887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:47.574918 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:47.575882 (+ 964us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:47.576229 (+ 347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:47.576233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:47.576234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:47.576238 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:47.576279 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:47.576293 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:47.576376 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:47.576552 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 20:47:47.576650 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":153,"threads_started":1}
I20260926 20:47:47.577155 21539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:47.577737 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:47:47.578745 21539 catalog_manager.cc:1347] Generated new cluster ID: fec926d0d99f4b05b8ca3994fa43f0e4
I20260926 20:47:47.578788 21539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:47:47.585098 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:47.583876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:34847 (local address 127.0.0.1:43495)
0926 20:47:47.583941 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:47.583943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:47.584072 (+ 129us) server_negotiation.cc:407] Connection header received
0926 20:47:47.584136 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:47.584137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:47.584143 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:47.584200 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:47.584846 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:47.584848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:47.584848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:47.584852 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:47.584867 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:47.584871 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:47.584927 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:47.584984 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 20:47:47.585000 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:47:47.589377 21539 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:47.589582 21539 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:47.595151 21539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a63996f9835d4a51a5248e71036c9a87: Generated new TSK 0
I20260926 20:47:47.595383 21539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:47.598361 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 21462
2026-09-26T20:47:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:47 dist-test-slave-4hd3 krb5kdc[21561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:47 dist-test-slave-4hd3 krb5kdc[21561](info): set up 2 sockets
Sep 26 20:47:47 dist-test-slave-4hd3 krb5kdc[21561](info): commencing operation
krb5kdc: starting...
W20260926 20:47:48.628074 393 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:48 dist-test-slave-4hd3 krb5kdc[21561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:47:48.745182 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40827
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:48.825642 21574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:48.825836 21574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:48.825858 21574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:48.827035 21574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:48.827085 21574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:48.827098 21574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:48.827111 21574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:48.827123 21574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:48.828645 21574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40827
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:48.828892 21574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:48.829129 21574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:48.834126 21582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:48.834309 21574 server_base.cc:1034] running on GCE node
W20260926 20:47:48.834133 21579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:48.834133 21580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:48.834656 21574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:48.834957 21574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:48.836133 21574 hybrid_clock.cc:648] HybridClock initialized: now 1790455668836118 us; error 29 us; skew 500 ppm
Sep 26 20:47:48 dist-test-slave-4hd3 krb5kdc[21561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:48.842267 21574 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:47:48.842738 21574 webserver.cc:460] Webserver started at http://127.0.0.1:34287/ using document root <none> and password file <none>
I20260926 20:47:48.842970 21574 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:48.843041 21574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:48.843204 21574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:48.844403 21574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "7c8758968dc042e1bf34f6cea07ee35a"
format_stamp: "Formatted at 2026-09-26 20:47:48 on dist-test-slave-4hd3"
server_key: "e5e45fd15def1b8cc2ae318d42325d4d"
server_key_iv: "2fca9a7075219c168fc60a9a4f33d525"
server_key_version: "encryptionkey@0"
I20260926 20:47:48.844746 21574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "7c8758968dc042e1bf34f6cea07ee35a"
format_stamp: "Formatted at 2026-09-26 20:47:48 on dist-test-slave-4hd3"
server_key: "e5e45fd15def1b8cc2ae318d42325d4d"
server_key_iv: "2fca9a7075219c168fc60a9a4f33d525"
server_key_version: "encryptionkey@0"
I20260926 20:47:48.846061 21574 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:47:48.846868 21589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:48.847091 21574 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:48.847167 21574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "7c8758968dc042e1bf34f6cea07ee35a"
format_stamp: "Formatted at 2026-09-26 20:47:48 on dist-test-slave-4hd3"
server_key: "e5e45fd15def1b8cc2ae318d42325d4d"
server_key_iv: "2fca9a7075219c168fc60a9a4f33d525"
server_key_version: "encryptionkey@0"
I20260926 20:47:48.847260 21574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:48.856827 21574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:48.859238 21574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:48.859390 21574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:48.863468 21574 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40827
I20260926 20:47:48.863507 21641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40827 every 8 connection(s)
I20260926 20:47:48.863942 21574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:48.864805 21642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:48.867053 21642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a: Bootstrap starting.
I20260926 20:47:48.867628 21642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:48.867931 21642 log.cc:824] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:48.868659 21642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a: No bootstrap required, opened a new log
I20260926 20:47:48.870225 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 21574
I20260926 20:47:48.870393 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:48.870275 21642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8758968dc042e1bf34f6cea07ee35a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40827 } }
I20260926 20:47:48.870445 21642 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:48.870508 21642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c8758968dc042e1bf34f6cea07ee35a, State: Initialized, Role: FOLLOWER
I20260926 20:47:48.870680 393 external_mini_cluster.cc:1428] Setting key cfce75fb77c531a6e8841ba768187767
I20260926 20:47:48.870700 21642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8758968dc042e1bf34f6cea07ee35a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40827 } }
I20260926 20:47:48.870790 21642 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:48.870828 21642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:48.870878 21642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:48.871874 21642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8758968dc042e1bf34f6cea07ee35a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40827 } }
I20260926 20:47:48.871991 21642 leader_election.cc:302] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c8758968dc042e1bf34f6cea07ee35a; no voters:
I20260926 20:47:48.872146 21642 leader_election.cc:288] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:48.872175 21647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:48.872278 21647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a [term 1 LEADER]: Becoming Leader. State: Replica: 7c8758968dc042e1bf34f6cea07ee35a, State: Running, Role: LEADER
I20260926 20:47:48.872413 21642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:48.872421 21647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8758968dc042e1bf34f6cea07ee35a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40827 } }
I20260926 20:47:48.872866 21649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c8758968dc042e1bf34f6cea07ee35a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8758968dc042e1bf34f6cea07ee35a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40827 } } }
I20260926 20:47:48.872882 21648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c8758968dc042e1bf34f6cea07ee35a. Latest consensus state: current_term: 1 leader_uuid: "7c8758968dc042e1bf34f6cea07ee35a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8758968dc042e1bf34f6cea07ee35a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40827 } } }
I20260926 20:47:48.872982 21649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:48.873013 21648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:48.873941 21656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:48.874585 21656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:48.876698 21656 catalog_manager.cc:1347] Generated new cluster ID: 96adac2804d0467497672948c44dc415
I20260926 20:47:48.876760 21656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:47:48 dist-test-slave-4hd3 krb5kdc[21561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:47:48.883411 21646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:48.871233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:43581 (local address 127.0.0.1:40827)
0926 20:47:48.871431 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:48.871442 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:48.871479 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:47:48.871600 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:48.871606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:48.871654 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:48.871880 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:48.872746 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:48.874358 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:48.875434 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:48.876127 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:48.876251 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:48.879437 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:48.879442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:48.879442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:48.879454 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:48.881621 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:48.882094 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:48.882097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:48.882097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:48.882222 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:48.882471 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:48.882472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:48.882473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:48.882802 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:48.882967 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:48.883094 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 20:47:48.883197 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":106,"threads_started":1}
I20260926 20:47:48.884475 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:47:48.900117 21646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:48.892066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:37281 (local address 127.0.0.1:40827)
0926 20:47:48.892105 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:48.892107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:48.892168 (+ 61us) server_negotiation.cc:407] Connection header received
0926 20:47:48.892240 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:48.892241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:48.892250 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:48.892318 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:48.892991 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:48.893835 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:48.894902 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:48.895239 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:48.895319 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:48.897163 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:48.897165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:48.897165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:48.897172 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:48.898838 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:48.899257 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:48.899258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:48.899258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:48.899311 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:48.899515 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:48.899515 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:48.899515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:48.899623 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:48.899707 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:48.899952 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 20:47:48.900023 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:47:48.906479 21656 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:48.907132 21656 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:48.914386 21656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c8758968dc042e1bf34f6cea07ee35a: Generated new TSK 0
I20260926 20:47:48.914589 21656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:48.922003 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 21574
W20260926 20:47:48.926369 21559 connection.cc:535] client connection to 127.0.0.1:40827 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40827): Connection reset by peer (error 104)
2026-09-26T20:47:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:48 dist-test-slave-4hd3 krb5kdc[21673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:48 dist-test-slave-4hd3 krb5kdc[21673](info): set up 2 sockets
Sep 26 20:47:48 dist-test-slave-4hd3 krb5kdc[21673](info): commencing operation
krb5kdc: starting...
W20260926 20:47:49.951920 393 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:50 dist-test-slave-4hd3 krb5kdc[21673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:47:50.071182 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44525
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:50.152385 21686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:50.152607 21686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:50.152645 21686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:50.153781 21686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:50.153847 21686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:50.153872 21686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:50.153898 21686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:50.153939 21686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:50.155465 21686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44525
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:50.155745 21686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:50.155967 21686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:50.160759 21691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:50.160809 21694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:50.160760 21692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:50.160776 21686 server_base.cc:1034] running on GCE node
I20260926 20:47:50.161214 21686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:50.161499 21686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:50.162667 21686 hybrid_clock.cc:648] HybridClock initialized: now 1790455670162653 us; error 32 us; skew 500 ppm
Sep 26 20:47:50 dist-test-slave-4hd3 krb5kdc[21673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:50.165455 21686 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:47:50.165906 21686 webserver.cc:460] Webserver started at http://127.0.0.1:41437/ using document root <none> and password file <none>
I20260926 20:47:50.166146 21686 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:50.166215 21686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:50.166352 21686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:50.167410 21686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "c47b89a46a1743c59dafec7e1fba591b"
format_stamp: "Formatted at 2026-09-26 20:47:50 on dist-test-slave-4hd3"
server_key: "8b761d07ae373dc192968628c6de270a"
server_key_iv: "090a2896803f1995b3566d762f8ecb25"
server_key_version: "encryptionkey@0"
I20260926 20:47:50.167837 21686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "c47b89a46a1743c59dafec7e1fba591b"
format_stamp: "Formatted at 2026-09-26 20:47:50 on dist-test-slave-4hd3"
server_key: "8b761d07ae373dc192968628c6de270a"
server_key_iv: "090a2896803f1995b3566d762f8ecb25"
server_key_version: "encryptionkey@0"
I20260926 20:47:50.169169 21686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:50.169961 21701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:50.170159 21686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:50.170257 21686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "c47b89a46a1743c59dafec7e1fba591b"
format_stamp: "Formatted at 2026-09-26 20:47:50 on dist-test-slave-4hd3"
server_key: "8b761d07ae373dc192968628c6de270a"
server_key_iv: "090a2896803f1995b3566d762f8ecb25"
server_key_version: "encryptionkey@0"
I20260926 20:47:50.170346 21686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:50.194496 21686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:50.196926 21686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:50.197103 21686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:50.201126 21686 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44525
I20260926 20:47:50.201165 21753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44525 every 8 connection(s)
I20260926 20:47:50.201524 21686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:50.202464 21754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:50.204706 21754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b: Bootstrap starting.
I20260926 20:47:50.205314 21754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:50.205600 21754 log.cc:824] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:50.205978 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 21686
I20260926 20:47:50.206110 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:50.206314 393 external_mini_cluster.cc:1428] Setting key a15c372d841d17ebb8bcac02ecf40d20
I20260926 20:47:50.206384 21754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b: No bootstrap required, opened a new log
I20260926 20:47:50.209031 21754 raft_consensus.cc:348] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47b89a46a1743c59dafec7e1fba591b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44525 } }
I20260926 20:47:50.209210 21754 raft_consensus.cc:374] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:50.209250 21754 raft_consensus.cc:729] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47b89a46a1743c59dafec7e1fba591b, State: Initialized, Role: FOLLOWER
I20260926 20:47:50.209375 21754 consensus_queue.cc:260] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47b89a46a1743c59dafec7e1fba591b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44525 } }
I20260926 20:47:50.209450 21754 raft_consensus.cc:388] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:50.209491 21754 raft_consensus.cc:482] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:50.209537 21754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:50.210210 21754 raft_consensus.cc:504] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47b89a46a1743c59dafec7e1fba591b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44525 } }
I20260926 20:47:50.210342 21754 leader_election.cc:302] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c47b89a46a1743c59dafec7e1fba591b; no voters:
I20260926 20:47:50.210573 21759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:50.210554 21754 leader_election.cc:288] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:50.210713 21759 raft_consensus.cc:686] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b [term 1 LEADER]: Becoming Leader. State: Replica: c47b89a46a1743c59dafec7e1fba591b, State: Running, Role: LEADER
I20260926 20:47:50.210842 21754 sys_catalog.cc:564] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:50.210857 21759 consensus_queue.cc:237] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47b89a46a1743c59dafec7e1fba591b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44525 } }
I20260926 20:47:50.211162 21759 sys_catalog.cc:455] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c47b89a46a1743c59dafec7e1fba591b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47b89a46a1743c59dafec7e1fba591b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44525 } } }
I20260926 20:47:50.211302 21759 sys_catalog.cc:458] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:50.211316 21760 sys_catalog.cc:455] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c47b89a46a1743c59dafec7e1fba591b. Latest consensus state: current_term: 1 leader_uuid: "c47b89a46a1743c59dafec7e1fba591b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47b89a46a1743c59dafec7e1fba591b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44525 } } }
I20260926 20:47:50.211570 21767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:50.212146 21767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:50.212572 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:50.206825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:37101 (local address 127.0.0.1:44525)
0926 20:47:50.206992 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:50.207001 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:50.207035 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:47:50.207675 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:50.207682 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:50.207784 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:50.207916 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:50.208181 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:50.209059 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:50.210178 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:50.210797 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:50.210932 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:50.211731 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:50.211736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:50.211736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:50.211936 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:50.211975 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:50.211992 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:50.212105 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:50.212289 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 20:47:50.212319 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260926 20:47:50.212860 21760 sys_catalog.cc:458] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:50.213572 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:47:50.213928 21767 catalog_manager.cc:1347] Generated new cluster ID: 232a118e486d41afb90a687d318ce565
I20260926 20:47:50.213979 21767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:47:50.222579 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:50.216112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:46871 (local address 127.0.0.1:44525)
0926 20:47:50.216146 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:50.216148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:50.219891 (+ 3743us) server_negotiation.cc:407] Connection header received
0926 20:47:50.219982 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:50.219983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:50.219993 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:50.220040 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:50.220193 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:50.220650 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:50.221375 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:50.221593 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:50.221684 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:50.222322 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:50.222324 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:50.222324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:50.222329 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:50.222342 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:50.222346 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:50.222402 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:50.222475 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 20:47:50.222484 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:47:50.229352 21767 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:50.229928 21767 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:50.234520 21767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c47b89a46a1743c59dafec7e1fba591b: Generated new TSK 0
I20260926 20:47:50.234691 21767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:50.237543 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 21686
W20260926 20:47:50.241770 21671 connection.cc:535] client connection to 127.0.0.1:44525 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44525): Connection reset by peer (error 104)
2026-09-26T20:47:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:50 dist-test-slave-4hd3 krb5kdc[21785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:50 dist-test-slave-4hd3 krb5kdc[21785](info): set up 2 sockets
Sep 26 20:47:50 dist-test-slave-4hd3 krb5kdc[21785](info): commencing operation
krb5kdc: starting...
W20260926 20:47:51.267992 393 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:51 dist-test-slave-4hd3 krb5kdc[21785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:47:51.380481 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34629
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:51.459278 21798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:51.459478 21798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:51.459501 21798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:51.460702 21798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:51.460748 21798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:51.460762 21798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:51.460775 21798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:51.460788 21798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:51.462215 21798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34629
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:51.462460 21798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:51.462697 21798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:51.467478 21806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:51.467607 21804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:51.467502 21803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:51.467816 21798 server_base.cc:1034] running on GCE node
I20260926 20:47:51.468075 21798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:51.468365 21798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:51.469530 21798 hybrid_clock.cc:648] HybridClock initialized: now 1790455671469508 us; error 41 us; skew 500 ppm
Sep 26 20:47:51 dist-test-slave-4hd3 krb5kdc[21785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:51.472080 21798 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:47:51.472510 21798 webserver.cc:460] Webserver started at http://127.0.0.1:45013/ using document root <none> and password file <none>
I20260926 20:47:51.472738 21798 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:51.472811 21798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:51.472972 21798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:51.474020 21798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "1e01817086d54fffab7f27a5e4d4409f"
format_stamp: "Formatted at 2026-09-26 20:47:51 on dist-test-slave-4hd3"
server_key: "68e47b263ba9b68af4c840ea2857ae74"
server_key_iv: "eca78d9bc14e4cba9c5c6960f8a1b46c"
server_key_version: "encryptionkey@0"
I20260926 20:47:51.474382 21798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "1e01817086d54fffab7f27a5e4d4409f"
format_stamp: "Formatted at 2026-09-26 20:47:51 on dist-test-slave-4hd3"
server_key: "68e47b263ba9b68af4c840ea2857ae74"
server_key_iv: "eca78d9bc14e4cba9c5c6960f8a1b46c"
server_key_version: "encryptionkey@0"
I20260926 20:47:51.475688 21798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:47:51.476449 21813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:51.476621 21798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:47:51.476718 21798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "1e01817086d54fffab7f27a5e4d4409f"
format_stamp: "Formatted at 2026-09-26 20:47:51 on dist-test-slave-4hd3"
server_key: "68e47b263ba9b68af4c840ea2857ae74"
server_key_iv: "eca78d9bc14e4cba9c5c6960f8a1b46c"
server_key_version: "encryptionkey@0"
I20260926 20:47:51.476818 21798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:51.480055 21798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:51.481827 21798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:51.481979 21798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:51.485910 21798 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34629
I20260926 20:47:51.485940 21865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34629 every 8 connection(s)
I20260926 20:47:51.486256 21798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:51.487213 21866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:51.489464 21866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f: Bootstrap starting.
I20260926 20:47:51.490047 21866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:51.490329 21866 log.cc:824] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:51.491031 21866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f: No bootstrap required, opened a new log
I20260926 20:47:51.492739 21866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e01817086d54fffab7f27a5e4d4409f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34629 } }
I20260926 20:47:51.492873 21866 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:51.492919 21866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e01817086d54fffab7f27a5e4d4409f, State: Initialized, Role: FOLLOWER
I20260926 20:47:51.493113 21866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e01817086d54fffab7f27a5e4d4409f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34629 } }
I20260926 20:47:51.493192 21866 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:51.493235 21866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:51.493283 21866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:51.493793 21866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e01817086d54fffab7f27a5e4d4409f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34629 } }
I20260926 20:47:51.493901 21866 leader_election.cc:302] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e01817086d54fffab7f27a5e4d4409f; no voters:
I20260926 20:47:51.494086 21866 leader_election.cc:288] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:51.494118 21869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:51.494300 21869 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f [term 1 LEADER]: Becoming Leader. State: Replica: 1e01817086d54fffab7f27a5e4d4409f, State: Running, Role: LEADER
I20260926 20:47:51.494355 21866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:51.494499 21869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e01817086d54fffab7f27a5e4d4409f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34629 } }
I20260926 20:47:51.494633 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 21798
I20260926 20:47:51.494746 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:51.494961 393 external_mini_cluster.cc:1428] Setting key 42ce510c11839ca0dee26ac0027d845e
I20260926 20:47:51.495393 21870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e01817086d54fffab7f27a5e4d4409f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e01817086d54fffab7f27a5e4d4409f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34629 } } }
I20260926 20:47:51.495496 21870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:51.495647 21871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e01817086d54fffab7f27a5e4d4409f. Latest consensus state: current_term: 1 leader_uuid: "1e01817086d54fffab7f27a5e4d4409f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e01817086d54fffab7f27a5e4d4409f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34629 } } }
I20260926 20:47:51.495797 21871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:51.496002 21880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:51.496551 21880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:51.497525 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:51.495482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:42593 (local address 127.0.0.1:34629)
0926 20:47:51.495897 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:51.495907 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:51.495944 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:47:51.496060 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:51.496067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:51.496097 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:51.496230 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:51.497064 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:51.497067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:51.497068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:51.497072 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:51.497116 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:51.497131 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:51.497208 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:51.497296 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 20:47:51.497342 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":74,"threads_started":1}
I20260926 20:47:51.498219 21880 catalog_manager.cc:1347] Generated new cluster ID: 76552bf0f6be47769c1586adcdc0d06b
I20260926 20:47:51.498315 21880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:47:51.498600 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:47:51.501157 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:51.500207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:56669 (local address 127.0.0.1:34629)
0926 20:47:51.500236 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:51.500238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:51.500254 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:47:51.500334 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:51.500335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:51.500340 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:51.500384 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:51.500953 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:51.500954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:51.500954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:51.500958 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:51.500971 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:51.500975 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:51.501023 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:51.501079 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 20:47:51.501091 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 20:47:51.513631 21880 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:51.513797 21880 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:51.518222 21880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e01817086d54fffab7f27a5e4d4409f: Generated new TSK 0
I20260926 20:47:51.518404 21880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:51.528105 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 21798
2026-09-26T20:47:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:51 dist-test-slave-4hd3 krb5kdc[21897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:51 dist-test-slave-4hd3 krb5kdc[21897](info): set up 2 sockets
Sep 26 20:47:51 dist-test-slave-4hd3 krb5kdc[21897](info): commencing operation
krb5kdc: starting...
W20260926 20:47:52.551837 393 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:52 dist-test-slave-4hd3 krb5kdc[21897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:47:52.670727 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:44835
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:44835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:52.751932 21910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:52.752177 21910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:52.752215 21910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:52.753381 21910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:52.753449 21910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:52.753475 21910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:52.753510 21910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:52.753535 21910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:52.754995 21910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:44835
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:44835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:52.755263 21910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:52.755482 21910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:47:52.760363 21910 server_base.cc:1034] running on GCE node
W20260926 20:47:52.760334 21915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:52.760317 21918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:52.760318 21916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:52.760805 21910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:52.761062 21910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:52.762205 21910 hybrid_clock.cc:648] HybridClock initialized: now 1790455672762195 us; error 26 us; skew 500 ppm
Sep 26 20:47:52 dist-test-slave-4hd3 krb5kdc[21897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:52.764664 21910 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:47:52.765158 21910 webserver.cc:460] Webserver started at http://127.0.98.126:43477/ using document root <none> and password file <none>
I20260926 20:47:52.765377 21910 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:52.765444 21910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:52.765596 21910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:52.766646 21910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "79b4500e69564b91a455e4c7051a2bf4"
format_stamp: "Formatted at 2026-09-26 20:47:52 on dist-test-slave-4hd3"
server_key: "f7dbb2415b3df63e20d0e6aad184aee8"
server_key_iv: "0662420fac2a9276a645693b1abdc194"
server_key_version: "encryptionkey@0"
I20260926 20:47:52.766999 21910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "79b4500e69564b91a455e4c7051a2bf4"
format_stamp: "Formatted at 2026-09-26 20:47:52 on dist-test-slave-4hd3"
server_key: "f7dbb2415b3df63e20d0e6aad184aee8"
server_key_iv: "0662420fac2a9276a645693b1abdc194"
server_key_version: "encryptionkey@0"
I20260926 20:47:52.768400 21910 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 20:47:52.769171 21925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:52.769383 21910 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:52.769488 21910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "79b4500e69564b91a455e4c7051a2bf4"
format_stamp: "Formatted at 2026-09-26 20:47:52 on dist-test-slave-4hd3"
server_key: "f7dbb2415b3df63e20d0e6aad184aee8"
server_key_iv: "0662420fac2a9276a645693b1abdc194"
server_key_version: "encryptionkey@0"
I20260926 20:47:52.769562 21910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:52.800943 21910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:52.803022 21910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:52.803189 21910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:52.807170 21910 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:44835
I20260926 20:47:52.807210 21977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:44835 every 8 connection(s)
I20260926 20:47:52.807582 21910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:52.808467 21978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:52.810714 21978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4: Bootstrap starting.
I20260926 20:47:52.811305 21978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:52.811595 21978 log.cc:824] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:52.812320 21978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4: No bootstrap required, opened a new log
I20260926 20:47:52.813900 21978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b4500e69564b91a455e4c7051a2bf4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44835 } }
I20260926 20:47:52.814038 21978 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:52.814061 21978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b4500e69564b91a455e4c7051a2bf4, State: Initialized, Role: FOLLOWER
I20260926 20:47:52.814260 21978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b4500e69564b91a455e4c7051a2bf4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44835 } }
I20260926 20:47:52.814329 21978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:52.814350 21978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:52.814396 21978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:52.814920 21978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b4500e69564b91a455e4c7051a2bf4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44835 } }
I20260926 20:47:52.815004 21978 leader_election.cc:302] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79b4500e69564b91a455e4c7051a2bf4; no voters:
I20260926 20:47:52.815181 21978 leader_election.cc:288] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:52.815239 21981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:52.815438 21978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:52.815440 21981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 79b4500e69564b91a455e4c7051a2bf4, State: Running, Role: LEADER
I20260926 20:47:52.815586 21981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b4500e69564b91a455e4c7051a2bf4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44835 } }
I20260926 20:47:52.815635 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 21910
I20260926 20:47:52.815764 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:52.815958 393 external_mini_cluster.cc:1428] Setting key ddf1986b7117dc140afacc80fbae84c2
I20260926 20:47:52.816131 21981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79b4500e69564b91a455e4c7051a2bf4. Latest consensus state: current_term: 1 leader_uuid: "79b4500e69564b91a455e4c7051a2bf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b4500e69564b91a455e4c7051a2bf4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44835 } } }
I20260926 20:47:52.816277 21981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:52.816306 21982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79b4500e69564b91a455e4c7051a2bf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b4500e69564b91a455e4c7051a2bf4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44835 } } }
I20260926 20:47:52.816390 21982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:52.816534 21991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:52.817070 21991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:52.818743 21991 catalog_manager.cc:1347] Generated new cluster ID: 7242566aa41b49b08bc80d8c5916bd8c
I20260926 20:47:52.818794 21991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:47:52 dist-test-slave-4hd3 krb5kdc[21897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:47:52.826867 21995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:52.816612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39585 (local address 127.0.98.126:44835)
0926 20:47:52.816808 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:52.816819 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:52.817010 (+ 191us) server_negotiation.cc:407] Connection header received
0926 20:47:52.817295 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:52.817303 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:52.817365 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:52.818174 (+ 809us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:52.818269 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:52.819380 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:52.820209 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:52.820646 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:52.820760 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:52.823094 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:52.823098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:52.823098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:52.823110 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:52.825351 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:52.825743 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:52.825746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:52.825747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:52.825850 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:52.826085 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:52.826086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:52.826087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:52.826322 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:52.826500 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:52.826596 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 20:47:52.826701 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260926 20:47:52.827803 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:47:52.835443 21995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:52.829206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46677 (local address 127.0.98.126:44835)
0926 20:47:52.829246 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:52.829248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:52.829391 (+ 143us) server_negotiation.cc:407] Connection header received
0926 20:47:52.829463 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:52.829464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:52.829473 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:52.829526 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:52.830025 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:52.830644 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:52.831347 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:52.831580 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:52.831655 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:52.832846 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:52.832848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:52.832848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:52.832855 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:52.834258 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:52.834631 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:52.834632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:52.834632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:52.834694 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:52.834916 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:52.834917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:52.834918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:52.835031 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:52.835158 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:52.835278 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 20:47:52.835344 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:47:52.851539 21991 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:52.852141 21991 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:52.858777 21991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79b4500e69564b91a455e4c7051a2bf4: Generated new TSK 0
I20260926 20:47:52.858989 21991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:52.861418 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 21910
W20260926 20:47:52.865345 21895 connection.cc:535] client connection to 127.0.98.126:44835 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.126:44835): Connection reset by peer (error 104)
2026-09-26T20:47:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:52 dist-test-slave-4hd3 krb5kdc[22009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:52 dist-test-slave-4hd3 krb5kdc[22009](info): set up 2 sockets
Sep 26 20:47:52 dist-test-slave-4hd3 krb5kdc[22009](info): commencing operation
krb5kdc: starting...
W20260926 20:47:53.891805 393 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:53 dist-test-slave-4hd3 krb5kdc[22009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:47:54.006527 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:44377
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:44377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:54.087924 22022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:54.088165 22022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:54.088204 22022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:54.089385 22022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:54.089453 22022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:54.089481 22022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:54.089514 22022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:54.089538 22022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:54.091084 22022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:44377
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:44377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:54.091364 22022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:54.091588 22022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:47:54.096496 22022 server_base.cc:1034] running on GCE node
W20260926 20:47:54.096442 22027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:54.096460 22028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:54.096433 22030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:54.096967 22022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:54.097268 22022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:54.098441 22022 hybrid_clock.cc:648] HybridClock initialized: now 1790455674098425 us; error 44 us; skew 500 ppm
Sep 26 20:47:54 dist-test-slave-4hd3 krb5kdc[22009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:54.104768 22022 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:47:54.105237 22022 webserver.cc:460] Webserver started at http://127.0.98.126:32789/ using document root <none> and password file <none>
I20260926 20:47:54.105471 22022 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:54.105542 22022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:54.105693 22022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:54.106712 22022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "9af81b4f3217475eb06701df862fd5c7"
format_stamp: "Formatted at 2026-09-26 20:47:54 on dist-test-slave-4hd3"
server_key: "029f4413948205ec79f6137259666861"
server_key_iv: "7ac241ad1957c264a26d55b6f1d76eb9"
server_key_version: "encryptionkey@0"
I20260926 20:47:54.107069 22022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "9af81b4f3217475eb06701df862fd5c7"
format_stamp: "Formatted at 2026-09-26 20:47:54 on dist-test-slave-4hd3"
server_key: "029f4413948205ec79f6137259666861"
server_key_iv: "7ac241ad1957c264a26d55b6f1d76eb9"
server_key_version: "encryptionkey@0"
I20260926 20:47:54.108487 22022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:47:54.109244 22037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:54.109493 22022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:47:54.109596 22022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "9af81b4f3217475eb06701df862fd5c7"
format_stamp: "Formatted at 2026-09-26 20:47:54 on dist-test-slave-4hd3"
server_key: "029f4413948205ec79f6137259666861"
server_key_iv: "7ac241ad1957c264a26d55b6f1d76eb9"
server_key_version: "encryptionkey@0"
I20260926 20:47:54.109670 22022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:54.123443 22022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:54.125347 22022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:54.125516 22022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:54.129592 22022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:44377
I20260926 20:47:54.129624 22089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:44377 every 8 connection(s)
I20260926 20:47:54.129961 22022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:54.130781 22090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:54.131371 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 22022
I20260926 20:47:54.131493 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:54.131676 393 external_mini_cluster.cc:1428] Setting key 28b56e39bea82fc653dc3958734c424b
I20260926 20:47:54.133723 22090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7: Bootstrap starting.
I20260926 20:47:54.134353 22090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:54.134661 22090 log.cc:824] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:54.135407 22090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7: No bootstrap required, opened a new log
I20260926 20:47:54.137307 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:54.132244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60861 (local address 127.0.98.126:44377)
0926 20:47:54.132462 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:54.132470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:54.132495 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:47:54.132890 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:54.132896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:54.132940 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:54.133054 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:54.133343 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:54.134255 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:54.135290 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:54.135816 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:54.135915 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:54.136665 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:54.136669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:54.136669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:54.136825 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:54.136853 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:54.136863 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:54.136932 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:54.137050 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 20:47:54.137072 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":87,"threads_started":1}
I20260926 20:47:54.137769 22090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af81b4f3217475eb06701df862fd5c7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44377 } }
I20260926 20:47:54.137946 22090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:54.138001 22090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9af81b4f3217475eb06701df862fd5c7, State: Initialized, Role: FOLLOWER
I20260926 20:47:54.138108 22090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af81b4f3217475eb06701df862fd5c7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44377 } }
I20260926 20:47:54.138211 22090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:54.138325 22090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:54.138329 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:47:54.138404 22090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:54.139084 22090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af81b4f3217475eb06701df862fd5c7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44377 } }
I20260926 20:47:54.139197 22090 leader_election.cc:302] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9af81b4f3217475eb06701df862fd5c7; no voters:
I20260926 20:47:54.139359 22090 leader_election.cc:288] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:54.139436 22096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:54.139596 22096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7 [term 1 LEADER]: Becoming Leader. State: Replica: 9af81b4f3217475eb06701df862fd5c7, State: Running, Role: LEADER
I20260926 20:47:54.139688 22096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af81b4f3217475eb06701df862fd5c7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44377 } }
I20260926 20:47:54.139771 22090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:54.140079 22096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9af81b4f3217475eb06701df862fd5c7. Latest consensus state: current_term: 1 leader_uuid: "9af81b4f3217475eb06701df862fd5c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af81b4f3217475eb06701df862fd5c7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44377 } } }
I20260926 20:47:54.140086 22100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9af81b4f3217475eb06701df862fd5c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af81b4f3217475eb06701df862fd5c7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44377 } } }
I20260926 20:47:54.140231 22100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:54.140436 22096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:54.140854 22105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:54.141355 22105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:54.143036 22105 catalog_manager.cc:1347] Generated new cluster ID: fea569eeab134613ac56e99478215041
I20260926 20:47:54.143086 22105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:47:54.143539 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:54.139866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53695 (local address 127.0.98.126:44377)
0926 20:47:54.139905 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:54.139906 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:54.140212 (+ 306us) server_negotiation.cc:407] Connection header received
0926 20:47:54.140316 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:54.140317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:54.140328 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:54.140361 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:54.140586 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:54.141330 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:54.142295 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:54.142575 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:54.142652 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:54.143309 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:54.143310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:54.143311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:54.143314 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:54.143328 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:54.143331 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:54.143392 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:54.143465 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 20:47:54.143474 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:47:54.151129 22105 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:54.152032 22105 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:54.157332 22105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9af81b4f3217475eb06701df862fd5c7: Generated new TSK 0
I20260926 20:47:54.157523 22105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:54.167541 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 22022
W20260926 20:47:54.171495 22007 connection.cc:535] client connection to 127.0.98.126:44377 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.126:44377): Connection reset by peer (error 104)
2026-09-26T20:47:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:54 dist-test-slave-4hd3 krb5kdc[22121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:54 dist-test-slave-4hd3 krb5kdc[22121](info): set up 2 sockets
Sep 26 20:47:54 dist-test-slave-4hd3 krb5kdc[22121](info): commencing operation
krb5kdc: starting...
W20260926 20:47:55.199640 393 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:55 dist-test-slave-4hd3 krb5kdc[22121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:47:55.314498 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40867
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:40867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:55.393275 22134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:55.393478 22134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:55.393501 22134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:55.394588 22134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:55.394624 22134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:55.394639 22134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:55.394651 22134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:55.394663 22134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:55.396071 22134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:40867
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:55.396272 22134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:55.396440 22134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:55.401368 22142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:55.401423 22134 server_base.cc:1034] running on GCE node
W20260926 20:47:55.401392 22140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:55.401391 22139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:55.401889 22134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:55.402225 22134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:55.403394 22134 hybrid_clock.cc:648] HybridClock initialized: now 1790455675403385 us; error 29 us; skew 500 ppm
Sep 26 20:47:55 dist-test-slave-4hd3 krb5kdc[22121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:55.405817 22134 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:47:55.406330 22134 webserver.cc:460] Webserver started at http://127.0.98.126:41919/ using document root <none> and password file <none>
I20260926 20:47:55.406563 22134 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:55.406631 22134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:55.406780 22134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:55.407881 22134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "20f0fcb2f9ac4313a659512a24ebd039"
format_stamp: "Formatted at 2026-09-26 20:47:55 on dist-test-slave-4hd3"
server_key: "abc98dc6b4d25b9fcfa57d5a380700f9"
server_key_iv: "c661c4a815f2af0fb137ac8d80856e9e"
server_key_version: "encryptionkey@0"
I20260926 20:47:55.408248 22134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "20f0fcb2f9ac4313a659512a24ebd039"
format_stamp: "Formatted at 2026-09-26 20:47:55 on dist-test-slave-4hd3"
server_key: "abc98dc6b4d25b9fcfa57d5a380700f9"
server_key_iv: "c661c4a815f2af0fb137ac8d80856e9e"
server_key_version: "encryptionkey@0"
I20260926 20:47:55.409543 22134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:55.410281 22149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:55.410490 22134 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:55.410596 22134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "20f0fcb2f9ac4313a659512a24ebd039"
format_stamp: "Formatted at 2026-09-26 20:47:55 on dist-test-slave-4hd3"
server_key: "abc98dc6b4d25b9fcfa57d5a380700f9"
server_key_iv: "c661c4a815f2af0fb137ac8d80856e9e"
server_key_version: "encryptionkey@0"
I20260926 20:47:55.410671 22134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:55.413992 22134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:55.415971 22134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:55.416121 22134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:55.420112 22134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:40867
I20260926 20:47:55.420159 22201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:40867 every 8 connection(s)
I20260926 20:47:55.420476 22134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:55.421433 22202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:55.423561 22202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039: Bootstrap starting.
I20260926 20:47:55.424149 22202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:55.424420 22202 log.cc:824] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:55.425060 22202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039: No bootstrap required, opened a new log
I20260926 20:47:55.426584 22202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f0fcb2f9ac4313a659512a24ebd039" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40867 } }
I20260926 20:47:55.426719 22202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:55.426741 22202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f0fcb2f9ac4313a659512a24ebd039, State: Initialized, Role: FOLLOWER
I20260926 20:47:55.426885 22202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f0fcb2f9ac4313a659512a24ebd039" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40867 } }
I20260926 20:47:55.426949 22202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:55.426970 22202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:55.426995 22202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:55.427518 22202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f0fcb2f9ac4313a659512a24ebd039" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40867 } }
I20260926 20:47:55.427623 22202 leader_election.cc:302] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20f0fcb2f9ac4313a659512a24ebd039; no voters:
I20260926 20:47:55.427824 22202 leader_election.cc:288] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:55.427879 22205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:55.428045 22205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039 [term 1 LEADER]: Becoming Leader. State: Replica: 20f0fcb2f9ac4313a659512a24ebd039, State: Running, Role: LEADER
I20260926 20:47:55.428076 22202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:55.428157 22205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f0fcb2f9ac4313a659512a24ebd039" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40867 } }
I20260926 20:47:55.428721 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 22134
I20260926 20:47:55.428761 22206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20f0fcb2f9ac4313a659512a24ebd039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f0fcb2f9ac4313a659512a24ebd039" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40867 } } }
I20260926 20:47:55.428836 22206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:55.428854 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:55.429028 393 external_mini_cluster.cc:1428] Setting key 81e3a7ec9ef871b5e58f5770122d2ad3
I20260926 20:47:55.429265 22207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20f0fcb2f9ac4313a659512a24ebd039. Latest consensus state: current_term: 1 leader_uuid: "20f0fcb2f9ac4313a659512a24ebd039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f0fcb2f9ac4313a659512a24ebd039" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40867 } } }
I20260926 20:47:55.429374 22207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039 [sys.catalog]: This master's current role is: LEADER
W20260926 20:47:55.430009 22220 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:47:55.430076 22220 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:47:55.430166 22221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:55.430337 22221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:55.431371 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:55.429700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39663 (local address 127.0.98.126:40867)
0926 20:47:55.429866 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:55.429893 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:55.429929 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:47:55.430043 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:55.430050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:55.430079 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:55.430487 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:55.430941 (+ 454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:55.430944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:55.430945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:55.430948 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:55.430980 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:55.430993 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:55.431056 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:55.431138 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 20:47:55.431188 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260926 20:47:55.431877 22221 catalog_manager.cc:1347] Generated new cluster ID: 69c92ff2820d46de90452504504cb67f
I20260926 20:47:55.431932 22221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:47:55.432215 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:47:55.437500 22221 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:55.437638 22221 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:55.438642 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:55.436997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54325 (local address 127.0.98.126:40867)
0926 20:47:55.437038 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:55.437040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:55.437399 (+ 359us) server_negotiation.cc:407] Connection header received
0926 20:47:55.437593 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:55.437594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:55.437600 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:55.437635 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:55.438409 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:55.438410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:55.438410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:55.438414 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:55.438430 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:55.438435 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:55.438496 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:55.438557 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 20:47:55.438569 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:47:55.440768 22221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20f0fcb2f9ac4313a659512a24ebd039: Generated new TSK 0
I20260926 20:47:55.440939 22221 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:55.442386 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 22134
2026-09-26T20:47:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1274 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:55 dist-test-slave-4hd3 krb5kdc[22233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:55 dist-test-slave-4hd3 krb5kdc[22233](info): set up 2 sockets
Sep 26 20:47:55 dist-test-slave-4hd3 krb5kdc[22233](info): commencing operation
krb5kdc: starting...
W20260926 20:47:56.475888 393 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:56 dist-test-slave-4hd3 krb5kdc[22233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:47:56.590745 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:42645
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:42645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:56.670775 22246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:56.671033 22246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:56.671075 22246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:56.672462 22246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:56.672549 22246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:56.672577 22246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:56.672626 22246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:56.672683 22246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:56.674350 22246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:42645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:42645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:56.674733 22246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:56.674984 22246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:56.679687 22251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:56.679682 22252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:56.679682 22254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:56.679919 22246 server_base.cc:1034] running on GCE node
I20260926 20:47:56.680244 22246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:56.680583 22246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:56.681746 22246 hybrid_clock.cc:648] HybridClock initialized: now 1790455676681727 us; error 39 us; skew 500 ppm
Sep 26 20:47:56 dist-test-slave-4hd3 krb5kdc[22233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:56.684293 22246 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:47:56.684778 22246 webserver.cc:460] Webserver started at http://127.0.98.126:34197/ using document root <none> and password file <none>
I20260926 20:47:56.685000 22246 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:56.685070 22246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:56.685232 22246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:56.686280 22246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "79ff1b115520429b89d6af9bef445a2d"
format_stamp: "Formatted at 2026-09-26 20:47:56 on dist-test-slave-4hd3"
server_key: "cacd87ab6df1c0f50cbdb2133531bcfc"
server_key_iv: "e1cd938cd01d07a6cd6cc9d878524637"
server_key_version: "encryptionkey@0"
I20260926 20:47:56.686672 22246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "79ff1b115520429b89d6af9bef445a2d"
format_stamp: "Formatted at 2026-09-26 20:47:56 on dist-test-slave-4hd3"
server_key: "cacd87ab6df1c0f50cbdb2133531bcfc"
server_key_iv: "e1cd938cd01d07a6cd6cc9d878524637"
server_key_version: "encryptionkey@0"
I20260926 20:47:56.688042 22246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:47:56.688803 22261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:56.688992 22246 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:56.689095 22246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "79ff1b115520429b89d6af9bef445a2d"
format_stamp: "Formatted at 2026-09-26 20:47:56 on dist-test-slave-4hd3"
server_key: "cacd87ab6df1c0f50cbdb2133531bcfc"
server_key_iv: "e1cd938cd01d07a6cd6cc9d878524637"
server_key_version: "encryptionkey@0"
I20260926 20:47:56.689172 22246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:56.710824 22246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:56.712994 22246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:56.713258 22246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:56.717813 22246 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:42645
I20260926 20:47:56.717906 22313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:42645 every 8 connection(s)
I20260926 20:47:56.718290 22246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:56.719189 22314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:56.721594 22314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d: Bootstrap starting.
I20260926 20:47:56.722198 22314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:56.722477 22314 log.cc:824] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:56.723250 22314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d: No bootstrap required, opened a new log
I20260926 20:47:56.724858 22314 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ff1b115520429b89d6af9bef445a2d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42645 } }
I20260926 20:47:56.725006 22314 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:56.725031 22314 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ff1b115520429b89d6af9bef445a2d, State: Initialized, Role: FOLLOWER
I20260926 20:47:56.725203 22314 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ff1b115520429b89d6af9bef445a2d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42645 } }
I20260926 20:47:56.725287 22314 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:56.725330 22314 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:56.725445 22314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:56.725523 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 22246
I20260926 20:47:56.725625 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:56.725800 393 external_mini_cluster.cc:1428] Setting key e0e7ad8147dbeadf269798391f1b96d6
I20260926 20:47:56.726177 22314 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ff1b115520429b89d6af9bef445a2d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42645 } }
I20260926 20:47:56.726593 22314 leader_election.cc:302] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79ff1b115520429b89d6af9bef445a2d; no voters:
I20260926 20:47:56.726789 22314 leader_election.cc:288] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:56.727123 22314 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:56.726806 22319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:56.727931 22319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d [term 1 LEADER]: Becoming Leader. State: Replica: 79ff1b115520429b89d6af9bef445a2d, State: Running, Role: LEADER
I20260926 20:47:56.728096 22319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ff1b115520429b89d6af9bef445a2d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42645 } }
I20260926 20:47:56.728514 22319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79ff1b115520429b89d6af9bef445a2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ff1b115520429b89d6af9bef445a2d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42645 } } }
I20260926 20:47:56.728667 22319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:56.729023 22319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79ff1b115520429b89d6af9bef445a2d. Latest consensus state: current_term: 1 leader_uuid: "79ff1b115520429b89d6af9bef445a2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ff1b115520429b89d6af9bef445a2d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42645 } } }
I20260926 20:47:56.729089 22334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:56.729157 22319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:56.729671 22334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:56.731197 22334 catalog_manager.cc:1347] Generated new cluster ID: a522e012368a4146964f543c13501133
I20260926 20:47:56.731251 22334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:47:56 dist-test-slave-4hd3 krb5kdc[22233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:47:56.738087 22318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:56.726251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42821 (local address 127.0.98.126:42645)
0926 20:47:56.726461 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:56.726472 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:56.726512 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:47:56.726642 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:56.726650 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:56.726708 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:56.726905 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:56.727705 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:56.729267 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:56.730449 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:56.731117 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:56.731219 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:56.734169 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:56.734173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:56.734174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:56.734185 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:56.736329 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:56.736739 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:56.736742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:56.736742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:56.736880 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:56.737089 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:56.737090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:56.737090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:56.737458 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:56.737642 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:56.737715 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 20:47:56.737860 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":114,"threads_started":1}
I20260926 20:47:56.739063 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:47:56.742699 22334 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:56.743381 22334 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:56.747555 22318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:56.740624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54493 (local address 127.0.98.126:42645)
0926 20:47:56.740667 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:56.740669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:56.740825 (+ 156us) server_negotiation.cc:407] Connection header received
0926 20:47:56.740854 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:56.740855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:56.740865 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:56.740923 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:56.741501 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:56.742257 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:56.743190 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:56.743516 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:56.743598 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:56.744976 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:56.744977 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:56.744977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:47:56.744984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:56.746493 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:56.746836 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:56.746837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:56.746837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:56.746881 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:47:56.747041 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:47:56.747041 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:47:56.747042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:47:56.747149 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:56.747209 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:56.747380 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 20:47:56.747461 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:47:56.751076 22334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79ff1b115520429b89d6af9bef445a2d: Generated new TSK 0
I20260926 20:47:56.751268 22334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:56.755891 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 22246
2026-09-26T20:47:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:56 dist-test-slave-4hd3 krb5kdc[22345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:56 dist-test-slave-4hd3 krb5kdc[22345](info): set up 2 sockets
Sep 26 20:47:56 dist-test-slave-4hd3 krb5kdc[22345](info): commencing operation
krb5kdc: starting...
W20260926 20:47:57.787894 393 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:57 dist-test-slave-4hd3 krb5kdc[22345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:47:57.899837 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36617
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:45681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:36617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:57.980291 22358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:57.980507 22358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:57.980530 22358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:57.981616 22358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:57.981652 22358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:57.981665 22358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:57.981678 22358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:57.981690 22358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:57.983135 22358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:45681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:36617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:57.983331 22358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:57.983582 22358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:57.988531 22364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:57.988560 22358 server_base.cc:1034] running on GCE node
W20260926 20:47:57.988531 22363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:57.988564 22366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:57.989034 22358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:57.989318 22358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:57.990484 22358 hybrid_clock.cc:648] HybridClock initialized: now 1790455677990478 us; error 34 us; skew 500 ppm
Sep 26 20:47:57 dist-test-slave-4hd3 krb5kdc[22345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:57.993033 22358 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:47:57.993403 22358 webserver.cc:460] Webserver started at http://127.0.98.126:33335/ using document root <none> and password file <none>
I20260926 20:47:57.993629 22358 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:57.993675 22358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:57.993813 22358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:57.994835 22358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "20607b04dff14388ae112637d19511f3"
format_stamp: "Formatted at 2026-09-26 20:47:57 on dist-test-slave-4hd3"
server_key: "6d84d32e963e21da299626b198c43211"
server_key_iv: "c4e23f989aaf1880486b518b3a00d577"
server_key_version: "encryptionkey@0"
I20260926 20:47:57.995186 22358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "20607b04dff14388ae112637d19511f3"
format_stamp: "Formatted at 2026-09-26 20:47:57 on dist-test-slave-4hd3"
server_key: "6d84d32e963e21da299626b198c43211"
server_key_iv: "c4e23f989aaf1880486b518b3a00d577"
server_key_version: "encryptionkey@0"
I20260926 20:47:57.996579 22358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:47:57.997283 22373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:57.997467 22358 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:47:57.997556 22358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "20607b04dff14388ae112637d19511f3"
format_stamp: "Formatted at 2026-09-26 20:47:57 on dist-test-slave-4hd3"
server_key: "6d84d32e963e21da299626b198c43211"
server_key_iv: "c4e23f989aaf1880486b518b3a00d577"
server_key_version: "encryptionkey@0"
I20260926 20:47:57.997627 22358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:58.017243 22358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:58.019587 22358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:58.019723 22358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:58.023396 22358 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:36617
I20260926 20:47:58.023437 22425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:36617 every 8 connection(s)
I20260926 20:47:58.023808 22358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:58.024132 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 22358
I20260926 20:47:58.024299 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:58.024513 393 external_mini_cluster.cc:1428] Setting key 47aef904bc140bf003bc0c9bb2ee183b
I20260926 20:47:58.024889 22426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:58.028136 22426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3: Bootstrap starting.
I20260926 20:47:58.028849 22426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:58.029245 22426 log.cc:824] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:58.029930 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:58.025036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42751 (local address 127.0.98.126:36617)
0926 20:47:58.025203 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:58.025212 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:58.025246 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:47:58.025658 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:58.025664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:58.025709 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:58.025819 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:58.026057 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:58.026909 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:58.028040 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:58.028493 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:58.028571 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:58.029268 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:58.029272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:58.029272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:58.029486 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:58.029515 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:58.029526 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:58.029602 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:58.029690 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 20:47:58.029758 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260926 20:47:58.030004 22426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3: No bootstrap required, opened a new log
I20260926 20:47:58.030908 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:47:58.032153 22426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20607b04dff14388ae112637d19511f3" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36617 } }
I20260926 20:47:58.032367 22426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:58.032435 22426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20607b04dff14388ae112637d19511f3, State: Initialized, Role: FOLLOWER
I20260926 20:47:58.032574 22426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20607b04dff14388ae112637d19511f3" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36617 } }
I20260926 20:47:58.032649 22426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:58.032670 22426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:58.032727 22426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:58.033265 22426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20607b04dff14388ae112637d19511f3" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36617 } }
I20260926 20:47:58.033382 22426 leader_election.cc:302] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20607b04dff14388ae112637d19511f3; no voters:
I20260926 20:47:58.033576 22426 leader_election.cc:288] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:58.033602 22436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:58.033721 22436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3 [term 1 LEADER]: Becoming Leader. State: Replica: 20607b04dff14388ae112637d19511f3, State: Running, Role: LEADER
I20260926 20:47:58.033846 22426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:58.033849 22436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20607b04dff14388ae112637d19511f3" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36617 } }
I20260926 20:47:58.034250 22436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20607b04dff14388ae112637d19511f3. Latest consensus state: current_term: 1 leader_uuid: "20607b04dff14388ae112637d19511f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20607b04dff14388ae112637d19511f3" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36617 } } }
I20260926 20:47:58.034358 22436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:58.034649 22438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20607b04dff14388ae112637d19511f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20607b04dff14388ae112637d19511f3" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36617 } } }
I20260926 20:47:58.034739 22438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3 [sys.catalog]: This master's current role is: LEADER
W20260926 20:47:58.035693 22452 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:47:58.035790 22452 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:47:58.035861 22443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:58.036049 22443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:58.036491 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:58.032350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33783 (local address 127.0.98.126:36617)
0926 20:47:58.032389 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:58.032390 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:58.032424 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:47:58.032503 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:58.032504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:58.032514 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:58.032568 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:58.032773 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:58.033266 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:58.034137 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:47:58.034441 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:47:58.034544 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:47:58.036238 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:58.036240 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:58.036240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:58.036245 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:58.036267 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:58.036272 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:58.036335 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:58.036389 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 20:47:58.036416 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:47:58.037685 22443 catalog_manager.cc:1347] Generated new cluster ID: 90aa5ea283ed48c48ccd6bc05ffcf495
I20260926 20:47:58.037750 22443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:47:58.049031 22443 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:58.049901 22443 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:58.057405 22443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20607b04dff14388ae112637d19511f3: Generated new TSK 0
I20260926 20:47:58.057631 22443 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:58.064622 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 22358
2026-09-26T20:47:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:58 dist-test-slave-4hd3 krb5kdc[22457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:58 dist-test-slave-4hd3 krb5kdc[22457](info): set up 2 sockets
Sep 26 20:47:58 dist-test-slave-4hd3 krb5kdc[22457](info): commencing operation
krb5kdc: starting...
W20260926 20:47:59.101229 393 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:47:59 dist-test-slave-4hd3 krb5kdc[22457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:47:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:47:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:47:59.234086 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:41617
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:41617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:47:59.313519 22470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:47:59.313701 22470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:47:59.313722 22470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:47:59.314770 22470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:47:59.314803 22470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:47:59.314817 22470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:47:59.314831 22470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:47:59.314842 22470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:47:59.316210 22470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:41617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:41617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:47:59.316385 22470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:47:59.316543 22470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:47:59.321327 22476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:47:59.321364 22475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:59.321396 22470 server_base.cc:1034] running on GCE node
W20260926 20:47:59.321360 22478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:47:59.321746 22470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:47:59.322027 22470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:47:59.323180 22470 hybrid_clock.cc:648] HybridClock initialized: now 1790455679323175 us; error 38 us; skew 500 ppm
Sep 26 20:47:59 dist-test-slave-4hd3 krb5kdc[22457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:47:59.332222 22470 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:47:59.332711 22470 webserver.cc:460] Webserver started at http://127.0.98.126:37235/ using document root <none> and password file <none>
I20260926 20:47:59.332949 22470 fs_manager.cc:359] Metadata directory not provided
I20260926 20:47:59.333019 22470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:47:59.333173 22470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:47:59.334220 22470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "025c78c66f3746d385b072eb17c36c45"
format_stamp: "Formatted at 2026-09-26 20:47:59 on dist-test-slave-4hd3"
server_key: "c5abc4b78bd1f0fb8ebab162e91e93fd"
server_key_iv: "196ece2674632f24197c435e1d69f77d"
server_key_version: "encryptionkey@0"
I20260926 20:47:59.334570 22470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "025c78c66f3746d385b072eb17c36c45"
format_stamp: "Formatted at 2026-09-26 20:47:59 on dist-test-slave-4hd3"
server_key: "c5abc4b78bd1f0fb8ebab162e91e93fd"
server_key_iv: "196ece2674632f24197c435e1d69f77d"
server_key_version: "encryptionkey@0"
I20260926 20:47:59.335978 22470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 20:47:59.336788 22485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:47:59.337005 22470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:47:59.337121 22470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "025c78c66f3746d385b072eb17c36c45"
format_stamp: "Formatted at 2026-09-26 20:47:59 on dist-test-slave-4hd3"
server_key: "c5abc4b78bd1f0fb8ebab162e91e93fd"
server_key_iv: "196ece2674632f24197c435e1d69f77d"
server_key_version: "encryptionkey@0"
I20260926 20:47:59.337200 22470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:47:59.340610 22470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:47:59.342912 22470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:47:59.343075 22470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:47:59.347262 22470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:41617
I20260926 20:47:59.347295 22537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:41617 every 8 connection(s)
I20260926 20:47:59.347625 22470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:47:59.348623 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 22470
I20260926 20:47:59.348745 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:47:59.348863 22538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:47:59.348984 393 external_mini_cluster.cc:1428] Setting key ef81ee9da1fbdad1a4909b48c334b9d7
I20260926 20:47:59.351398 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:59.349475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35973 (local address 127.0.98.126:41617)
0926 20:47:59.349669 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:59.349690 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:59.349714 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:47:59.350244 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:59.350251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:59.350275 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:59.350388 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:59.351020 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:59.351024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:59.351024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:59.351028 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:59.351065 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:59.351075 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:59.351140 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:59.351235 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 20:47:59.351266 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":99,"threads_started":1}
I20260926 20:47:59.352331 22538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45: Bootstrap starting.
I20260926 20:47:59.352496 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:47:59.352967 22538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45: Neither blocks nor log segments found. Creating new log.
I20260926 20:47:59.353245 22538 log.cc:824] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45: Log is configured to *not* fsync() on all Append() calls
I20260926 20:47:59.354010 22538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45: No bootstrap required, opened a new log
I20260926 20:47:59.354822 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:47:59.353907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44557 (local address 127.0.98.126:41617)
0926 20:47:59.353947 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:47:59.353948 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:47:59.354024 (+ 76us) server_negotiation.cc:407] Connection header received
0926 20:47:59.354100 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:47:59.354100 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:47:59.354105 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:47:59.354162 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:47:59.354660 (+ 498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:47:59.354661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:47:59.354661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:47:59.354664 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:47:59.354676 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:47:59.354679 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:47:59.354721 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 20:47:59.354773 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 20:47:59.354783 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:47:59.356354 22538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025c78c66f3746d385b072eb17c36c45" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41617 } }
I20260926 20:47:59.356487 22538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:47:59.356508 22538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025c78c66f3746d385b072eb17c36c45, State: Initialized, Role: FOLLOWER
I20260926 20:47:59.356654 22538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025c78c66f3746d385b072eb17c36c45" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41617 } }
I20260926 20:47:59.356737 22538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:47:59.356781 22538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:47:59.356829 22538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:47:59.357352 22538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025c78c66f3746d385b072eb17c36c45" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41617 } }
I20260926 20:47:59.357434 22538 leader_election.cc:302] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 025c78c66f3746d385b072eb17c36c45; no voters:
I20260926 20:47:59.357649 22538 leader_election.cc:288] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:47:59.357741 22548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:47:59.357887 22538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:47:59.357913 22548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45 [term 1 LEADER]: Becoming Leader. State: Replica: 025c78c66f3746d385b072eb17c36c45, State: Running, Role: LEADER
I20260926 20:47:59.357988 22548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025c78c66f3746d385b072eb17c36c45" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41617 } }
I20260926 20:47:59.358383 22550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 025c78c66f3746d385b072eb17c36c45. Latest consensus state: current_term: 1 leader_uuid: "025c78c66f3746d385b072eb17c36c45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025c78c66f3746d385b072eb17c36c45" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41617 } } }
I20260926 20:47:59.358503 22550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:59.358374 22549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "025c78c66f3746d385b072eb17c36c45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025c78c66f3746d385b072eb17c36c45" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41617 } } }
I20260926 20:47:59.358734 22549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45 [sys.catalog]: This master's current role is: LEADER
I20260926 20:47:59.358938 22557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:47:59.359396 22557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:47:59.361028 22557 catalog_manager.cc:1347] Generated new cluster ID: 71a069352c7149c8ad1d59c46f124799
I20260926 20:47:59.361090 22557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:47:59.378774 22557 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:47:59.378973 22557 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:47:59.383869 22557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 025c78c66f3746d385b072eb17c36c45: Generated new TSK 0
I20260926 20:47:59.384057 22557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:47:59.391403 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 22470
2026-09-26T20:47:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1326 ms)
[----------] 18 tests from AuthTokenIssuingTest (23579 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:47:59 dist-test-slave-4hd3 krb5kdc[22569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:47:59 dist-test-slave-4hd3 krb5kdc[22569](info): set up 2 sockets
Sep 26 20:47:59 dist-test-slave-4hd3 krb5kdc[22569](info): commencing operation
krb5kdc: starting...
W20260926 20:48:00.423843 393 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:48:00 dist-test-slave-4hd3 krb5kdc[22569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:48:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:48:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:48:00.539017 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33059
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:33059,127.0.98.125:46427,127.0.98.124:37365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:48:00.618558 22582 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:48:00.618755 22582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:00.618772 22582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:00.618794 22582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:00.619940 22582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:48:00.619980 22582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:00.619994 22582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:00.620007 22582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:48:00.620064 22582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:48:00.621490 22582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:33059,127.0.98.125:46427,127.0.98.124:37365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:00.621726 22582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:00.621956 22582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:00.626719 22590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:00.626736 22588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:00.626746 22587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:00.626827 22582 server_base.cc:1034] running on GCE node
I20260926 20:48:00.627190 22582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:00.627527 22582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:00.628706 22582 hybrid_clock.cc:648] HybridClock initialized: now 1790455680628684 us; error 44 us; skew 500 ppm
Sep 26 20:48:00 dist-test-slave-4hd3 krb5kdc[22569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:00.631197 22582 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:48:00.631693 22582 webserver.cc:460] Webserver started at http://127.0.98.126:44677/ using document root <none> and password file <none>
I20260926 20:48:00.631973 22582 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:00.632019 22582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:00.632110 22582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:00.633111 22582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "5c85d9a522b645dfa65101f943d78e08"
format_stamp: "Formatted at 2026-09-26 20:48:00 on dist-test-slave-4hd3"
server_key: "7f4246717c5e8f5c41ad7ce5611860c5"
server_key_iv: "c1cb270f08d3c8cd2d689de31fe5aa2c"
server_key_version: "encryptionkey@0"
I20260926 20:48:00.633487 22582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "5c85d9a522b645dfa65101f943d78e08"
format_stamp: "Formatted at 2026-09-26 20:48:00 on dist-test-slave-4hd3"
server_key: "7f4246717c5e8f5c41ad7ce5611860c5"
server_key_iv: "c1cb270f08d3c8cd2d689de31fe5aa2c"
server_key_version: "encryptionkey@0"
I20260926 20:48:00.634745 22582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:48:00.635448 22597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:00.635663 22582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:48:00.635769 22582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "5c85d9a522b645dfa65101f943d78e08"
format_stamp: "Formatted at 2026-09-26 20:48:00 on dist-test-slave-4hd3"
server_key: "7f4246717c5e8f5c41ad7ce5611860c5"
server_key_iv: "c1cb270f08d3c8cd2d689de31fe5aa2c"
server_key_version: "encryptionkey@0"
I20260926 20:48:00.635845 22582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:00.646762 22582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:00.648839 22582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:00.648993 22582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:00.653287 22582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:33059
I20260926 20:48:00.653333 22649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:33059 every 8 connection(s)
I20260926 20:48:00.653723 22582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:48:00.654822 22650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:00.656520 22650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:48:00 dist-test-slave-4hd3 krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:48:00.663672 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 22582
I20260926 20:48:00.663836 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:48:00.664031 393 external_mini_cluster.cc:1428] Setting key 55686c5b5674a5766b8756cf4b324aef
I20260926 20:48:00.666195 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.657221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33059 (local address 127.0.0.1:50620)
0926 20:48:00.657443 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.657482 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:00.657950 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:00.658269 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:00.658271 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:00.658540 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:00.658693 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:00.658697 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.659711 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.659712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:00.660411 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:00.660412 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.660912 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.660913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:00.660966 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:00.661518 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:00.661526 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:00.662852 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:00.664704 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:00.664710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:00.664712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:00.664985 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:00.665207 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:00.665209 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:00.665209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:00.665327 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:00.665788 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:00.665793 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:00.665844 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 20:48:00.665919 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:48:00.666037 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":96,"threads_started":1}
I20260926 20:48:00.666277 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.657257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50620 (local address 127.0.98.126:33059)
0926 20:48:00.658084 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:00.658087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:00.658098 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:48:00.658155 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:00.658156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:00.658171 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:00.658233 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:00.658789 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.659623 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.660507 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.660819 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.660901 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:00.662935 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:00.662942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:00.662942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:00.662948 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:00.664621 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:00.665065 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:00.665067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:00.665067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:00.665141 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:00.665427 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:00.665428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:00.665429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:00.665715 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:00.665803 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:00.665982 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 20:48:00.666126 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":807,"thread_start_us":77,"threads_started":1}
I20260926 20:48:00.667178 22650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:48:00.667516 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.667398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:46427 (local address 127.0.0.1:50614)
0926 20:48:00.667450 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.667499 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 20:48:00.667644 22600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46427: connect: Connection refused (error 111)
I20260926 20:48:00.668095 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.667992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:46427 (local address 127.0.0.1:50628)
0926 20:48:00.668050 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.668074 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 20:48:00.668205 22650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:46427: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46427: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.125.keytab.
Entry for principal kudu/127.0.98.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.125.keytab.
I20260926 20:48:00.686955 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:46427
--webserver_interface=127.0.98.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.125.keytab
--principal=kudu/127.0.98.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:33059,127.0.98.125:46427,127.0.98.124:37365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:48:00.717497 22650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } attempt: 1
I20260926 20:48:00.718112 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.717930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:46427 (local address 127.0.0.1:50632)
0926 20:48:00.718030 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.718079 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 20:48:00.718608 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.718506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:46427 (local address 127.0.0.1:50646)
0926 20:48:00.718556 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.718586 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 20:48:00.718726 22650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:46427: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46427: connect: Connection refused (error 111)
W20260926 20:48:00.768006 22657 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:48:00.768204 22657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:00.768222 22657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:00.768241 22657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:00.769377 22657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:48:00.769421 22657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:00.769434 22657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:00.769447 22657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:48:00.769460 22657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:48:00.770915 22657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:33059,127.0.98.125:46427,127.0.98.124:37365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.125.keytab
--principal=kudu/127.0.98.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:46427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:00.771152 22657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:00.771380 22657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:00.776021 22665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:00.776042 22663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:00.776039 22662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:00.776197 22657 server_base.cc:1034] running on GCE node
I20260926 20:48:00.776450 22657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:00.776731 22657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:00.777900 22657 hybrid_clock.cc:648] HybridClock initialized: now 1790455680777893 us; error 36 us; skew 500 ppm
Sep 26 20:48:00 dist-test-slave-4hd3 krb5kdc[22569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:00.780378 22657 init.cc:377] Logged in from keytab as kudu/127.0.98.125@KRBTEST.COM (short username kudu)
I20260926 20:48:00.780804 22657 webserver.cc:460] Webserver started at http://127.0.98.125:37445/ using document root <none> and password file <none>
I20260926 20:48:00.781039 22657 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:00.781108 22657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:00.781260 22657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:00.782289 22657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-1/data/instance:
uuid: "e0737988557c41199caac028484016ea"
format_stamp: "Formatted at 2026-09-26 20:48:00 on dist-test-slave-4hd3"
server_key: "7aaf02de74072fbaef895f9d886e5097"
server_key_iv: "aeecc5f3330bd7a2958bb6a69d4fdadc"
server_key_version: "encryptionkey@0"
I20260926 20:48:00.782640 22657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-1/wal/instance:
uuid: "e0737988557c41199caac028484016ea"
format_stamp: "Formatted at 2026-09-26 20:48:00 on dist-test-slave-4hd3"
server_key: "7aaf02de74072fbaef895f9d886e5097"
server_key_iv: "aeecc5f3330bd7a2958bb6a69d4fdadc"
server_key_version: "encryptionkey@0"
I20260926 20:48:00.783972 22657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:48:00.784686 22672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:00.784843 22657 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:48:00.784943 22657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-1/wal
uuid: "e0737988557c41199caac028484016ea"
format_stamp: "Formatted at 2026-09-26 20:48:00 on dist-test-slave-4hd3"
server_key: "7aaf02de74072fbaef895f9d886e5097"
server_key_iv: "aeecc5f3330bd7a2958bb6a69d4fdadc"
server_key_version: "encryptionkey@0"
I20260926 20:48:00.785023 22657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:00.786897 22650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } attempt: 2
I20260926 20:48:00.787259 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.787141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:46427 (local address 127.0.0.1:50652)
0926 20:48:00.787197 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.787230 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 20:48:00.787606 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.787516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:46427 (local address 127.0.0.1:50666)
0926 20:48:00.787559 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.787582 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 20:48:00.787688 22650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:46427: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46427: connect: Connection refused (error 111)
I20260926 20:48:00.813581 22657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:00.816941 22657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:00.817099 22657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:00.821441 22657 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.125:46427
I20260926 20:48:00.821491 22724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.125:46427 every 8 connection(s)
I20260926 20:48:00.821834 22657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-1/data/info.pb
I20260926 20:48:00.822806 22725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:00.824419 22725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:48:00 dist-test-slave-4hd3 krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:48:00.831544 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 22657
I20260926 20:48:00.831666 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-1/wal/instance
I20260926 20:48:00.831916 393 external_mini_cluster.cc:1428] Setting key 508528f45e2d0590c5a375b7a2447abd
I20260926 20:48:00.833236 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.825160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50622 (local address 127.0.98.126:33059)
0926 20:48:00.825238 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:00.825241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:00.825416 (+ 175us) server_negotiation.cc:407] Connection header received
0926 20:48:00.825898 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:00.825900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:00.825915 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:00.825968 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:00.826544 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.827037 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.828056 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.828281 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.828366 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:00.830339 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:00.830341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:00.830341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:00.830350 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:00.831817 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:00.832257 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:00.832258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:00.832259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:00.832318 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:00.832554 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:00.832555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:00.832556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:00.832698 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:00.832765 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:00.833003 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 20:48:00.833093 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:48:00.833250 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.825137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33059 (local address 127.0.0.1:50622)
0926 20:48:00.825303 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.825335 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:00.825788 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:00.826039 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:00.826042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:00.826271 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:00.826453 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:00.826457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.827129 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.827130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:00.827959 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:00.827962 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.828395 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.828396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:00.828514 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:00.828943 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:00.828951 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:00.830245 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:00.831900 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:00.831906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:00.831908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:00.832184 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:00.832392 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:00.832393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:00.832393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:00.832491 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:00.832760 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:00.832764 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:00.832879 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 20:48:00.832956 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:48:00.833074 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260926 20:48:00.833833 22725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:48:00 dist-test-slave-4hd3 krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.124@KRBTEST.COM; defaulting to no policy
I20260926 20:48:00.841812 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.834107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:46427 (local address 127.0.0.1:50674)
0926 20:48:00.834314 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.834371 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:00.834434 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:00.834566 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:00.834567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:00.834717 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:00.834775 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:00.834777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.835602 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.835603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:00.836507 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:00.836510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.837003 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.837004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:00.837092 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:00.837527 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:00.837533 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:00.838863 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:00.840749 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:00.840750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:00.840750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:00.840919 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:00.841147 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:00.841147 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:00.841148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:00.841202 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:00.841495 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:00.841495 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:00.841533 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 20:48:00.841597 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 20:48:00.841696 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260926 20:48:00.841869 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.834176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.0.98.125:46427)
0926 20:48:00.834324 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:00.834328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:00.834424 (+ 96us) server_negotiation.cc:407] Connection header received
0926 20:48:00.834490 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:00.834492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:00.834502 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:00.834561 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:00.834851 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.835517 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.836607 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.836925 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.836979 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:00.838979 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:00.838994 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:00.838994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:00.839000 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:00.840652 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:00.841010 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:00.841014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:00.841014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:00.841071 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:00.841267 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:00.841268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:00.841269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:00.841413 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:00.841503 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:00.841635 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 20:48:00.841746 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.124@KRBTEST.COM" created.
I20260926 20:48:00.842600 22725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:48:00.842948 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.842817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:37365 (local address 127.0.0.1:40080)
0926 20:48:00.842864 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.842919 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:37365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 20:48:00.843108 22674 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.124:37365: connect: Connection refused (error 111)
I20260926 20:48:00.843575 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.843465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:37365 (local address 127.0.0.1:40082)
0926 20:48:00.843515 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.843544 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:37365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 20:48:00.843747 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:37365: Network error: Client connection negotiation failed: client connection to 127.0.98.124:37365: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.124.keytab.
Entry for principal kudu/127.0.98.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.124.keytab.
I20260926 20:48:00.857987 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:37365
--webserver_interface=127.0.98.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.124.keytab
--principal=kudu/127.0.98.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:33059,127.0.98.125:46427,127.0.98.124:37365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:48:00.878849 22650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } attempt: 3
Sep 26 20:48:00 dist-test-slave-4hd3 krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260926 20:48:00.885803 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.879237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:46427 (local address 127.0.0.1:50684)
0926 20:48:00.879300 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.879310 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:00.879367 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:00.879581 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:00.879582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:00.879798 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:00.879868 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:00.879871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.880565 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.880566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:00.881106 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:00.881108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.881500 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.881501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:00.881578 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:00.882040 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:00.882046 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:00.883227 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:00.884790 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:00.884791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:00.884791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:00.884965 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:00.885174 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:00.885175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:00.885175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:00.885237 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:00.885478 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:00.885479 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:00.885524 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 20:48:00.885611 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 20:48:00.885706 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 20:48:00.885803 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.879323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50684 (local address 127.0.98.125:46427)
0926 20:48:00.879388 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:00.879390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:00.879400 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:48:00.879476 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:00.879477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:00.879491 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:00.879556 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:00.879951 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.880468 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.881211 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.881423 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.881504 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:00.883315 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:00.883317 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:00.883317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:00.883325 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:00.884704 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:00.885041 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:00.885042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:00.885043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:00.885098 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:00.885298 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:00.885299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:00.885300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:00.885411 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:00.885485 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:00.885636 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 20:48:00.885706 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:48:00.886276 22650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:48:00.886611 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.886513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:37365 (local address 127.0.0.1:40092)
0926 20:48:00.886565 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.886593 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:37365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 20:48:00.886922 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.886839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:37365 (local address 127.0.0.1:40106)
0926 20:48:00.886880 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.886903 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:37365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 20:48:00.887013 22650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:37365: Network error: Client connection negotiation failed: client connection to 127.0.98.124:37365: connect: Connection refused (error 111)
I20260926 20:48:00.892951 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } attempt: 1
I20260926 20:48:00.893311 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.893211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:37365 (local address 127.0.0.1:40114)
0926 20:48:00.893261 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.893290 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:37365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 20:48:00.893587 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.893536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:37365 (local address 127.0.0.1:40118)
0926 20:48:00.893557 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.893576 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:37365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260926 20:48:00.893680 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:37365: Network error: Client connection negotiation failed: client connection to 127.0.98.124:37365: connect: Connection refused (error 111)
I20260926 20:48:00.918147 22650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } attempt: 1
I20260926 20:48:00.918565 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.918441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:37365 (local address 127.0.0.1:40130)
0926 20:48:00.918498 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.918537 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:37365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 20:48:00.918897 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.918788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:37365 (local address 127.0.0.1:40134)
0926 20:48:00.918831 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.918854 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:37365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 20:48:00.918989 22650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:37365: Network error: Client connection negotiation failed: client connection to 127.0.98.124:37365: connect: Connection refused (error 111)
W20260926 20:48:00.942638 22732 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:48:00.942898 22732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:00.942929 22732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:00.942968 22732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:00.944664 22732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:48:00.944727 22732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:00.944751 22732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:00.944773 22732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:48:00.944801 22732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:48:00.946273 22732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:33059,127.0.98.125:46427,127.0.98.124:37365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/krb5kdc/kudu_127.0.98.124.keytab
--principal=kudu/127.0.98.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:37365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:00.946555 22732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:00.946789 22732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:00.951504 22737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:00.951517 22740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:00.951588 22738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:00.951876 22732 server_base.cc:1034] running on GCE node
I20260926 20:48:00.952135 22732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:00.952502 22732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:00.953686 22732 hybrid_clock.cc:648] HybridClock initialized: now 1790455680953675 us; error 29 us; skew 500 ppm
Sep 26 20:48:00 dist-test-slave-4hd3 krb5kdc[22569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:00.956249 22732 init.cc:377] Logged in from keytab as kudu/127.0.98.124@KRBTEST.COM (short username kudu)
I20260926 20:48:00.956772 22732 webserver.cc:460] Webserver started at http://127.0.98.124:35573/ using document root <none> and password file <none>
I20260926 20:48:00.956995 22732 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:00.957064 22732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:00.957217 22732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:00.958303 22732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-2/data/instance:
uuid: "ece21816afef41a1a4ca95782f0c5596"
format_stamp: "Formatted at 2026-09-26 20:48:00 on dist-test-slave-4hd3"
server_key: "1a22c6fc671863e11b79ecc39dc84d3d"
server_key_iv: "41037634ed5a28de352fcb89cac91971"
server_key_version: "encryptionkey@0"
I20260926 20:48:00.958673 22732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-2/wal/instance:
uuid: "ece21816afef41a1a4ca95782f0c5596"
format_stamp: "Formatted at 2026-09-26 20:48:00 on dist-test-slave-4hd3"
server_key: "1a22c6fc671863e11b79ecc39dc84d3d"
server_key_iv: "41037634ed5a28de352fcb89cac91971"
server_key_version: "encryptionkey@0"
I20260926 20:48:00.960035 22732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 20:48:00.960798 22747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:00.960958 22732 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:48:00.961062 22732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-2/wal
uuid: "ece21816afef41a1a4ca95782f0c5596"
format_stamp: "Formatted at 2026-09-26 20:48:00 on dist-test-slave-4hd3"
server_key: "1a22c6fc671863e11b79ecc39dc84d3d"
server_key_iv: "41037634ed5a28de352fcb89cac91971"
server_key_version: "encryptionkey@0"
I20260926 20:48:00.961136 22732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:00.961814 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } attempt: 2
I20260926 20:48:00.962217 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.962095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:37365 (local address 127.0.0.1:40138)
0926 20:48:00.962153 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.962190 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:37365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 20:48:00.962558 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.962434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:37365 (local address 127.0.0.1:40150)
0926 20:48:00.962510 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.962536 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:37365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 20:48:00.962641 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:37365: Network error: Client connection negotiation failed: client connection to 127.0.98.124:37365: connect: Connection refused (error 111)
I20260926 20:48:00.977839 22732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:00.980197 22732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:00.980307 22732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:00.984335 22732 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.124:37365
I20260926 20:48:00.984367 22799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.124:37365 every 8 connection(s)
I20260926 20:48:00.984740 22732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-2/data/info.pb
I20260926 20:48:00.985716 22800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:00.987216 22800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:48:00 dist-test-slave-4hd3 krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:48:00.993800 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 22732
I20260926 20:48:00.993911 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455416624047-393-0/minicluster-data/master-2/wal/instance
I20260926 20:48:00.994125 393 external_mini_cluster.cc:1428] Setting key 3008ecd64d3249cb3153c6e9b7e26717
I20260926 20:48:00.994144 22650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } attempt: 2
I20260926 20:48:00.996706 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.987940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50630 (local address 127.0.98.126:33059)
0926 20:48:00.988019 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:00.988021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:00.988194 (+ 173us) server_negotiation.cc:407] Connection header received
0926 20:48:00.988625 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:00.988626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:00.988637 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:00.988682 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:00.989384 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.989836 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.990771 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.990959 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.991016 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:00.993223 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:00.993225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:00.993225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:00.993232 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:00.994991 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:00.995592 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:00.995593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:00.995593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:00.995658 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:00.995969 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:00.995970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:00.995970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:00.996100 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:00.996170 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:00.996531 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 20:48:00.996613 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:48:00.996968 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.987903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33059 (local address 127.0.0.1:50630)
0926 20:48:00.988079 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.988115 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:00.988547 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:00.988750 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:00.988754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:00.989090 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:00.989281 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:00.989286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.989910 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.989910 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:00.990696 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:00.990698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.991046 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.991047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:00.991135 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:00.991654 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:00.991662 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:00.993140 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:00.995128 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:00.995134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:00.995135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:00.995504 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:00.995758 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:00.995759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:00.995760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:00.995893 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:00.996175 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:00.996179 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:00.996363 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 20:48:00.996480 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 20:48:00.996631 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260926 20:48:00.997701 22800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:48:00 dist-test-slave-4hd3 krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
Sep 26 20:48:00 dist-test-slave-4hd3 krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
Sep 26 20:48:00 dist-test-slave-4hd3 krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260926 20:48:01.003574 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.994468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40158 (local address 127.0.98.124:37365)
0926 20:48:00.994629 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:00.994633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:00.994654 (+ 21us) server_negotiation.cc:407] Connection header received
0926 20:48:00.994675 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:00.994678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:00.994693 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:00.994934 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:00.995297 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.996405 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.997381 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.997884 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.997950 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:01.000164 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:01.000172 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:01.000173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:01.000180 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:01.002211 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.002633 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.002636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.002637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.002694 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.002937 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.002938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.002938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.003152 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:01.003237 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:01.003408 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 20:48:01.003482 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260926 20:48:01.003576 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.994422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:37365 (local address 127.0.0.1:40158)
0926 20:48:00.994471 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.994482 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:00.994519 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:00.994816 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:00.994816 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:00.995153 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:00.995208 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:00.995210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.996506 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.996507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:00.997263 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:00.997265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.998004 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.998005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:00.998058 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:00.998427 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:00.998431 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:01.000082 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:01.002310 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:01.002311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:01.002311 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:01.002526 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:01.002787 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:01.002789 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:01.002789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:01.002858 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:01.003219 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:01.003220 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:01.003269 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 20:48:01.003351 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:48:01.003453 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:48:01.004968 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.994657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40154 (local address 127.0.98.124:37365)
0926 20:48:00.994797 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:00.994799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:00.994805 (+ 6us) server_negotiation.cc:407] Connection header received
0926 20:48:00.994820 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:00.994821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:00.994828 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:00.994902 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:00.995620 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.996389 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.997466 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.997880 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.997969 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:01.000833 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:01.000835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:01.000835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:01.000842 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:01.003398 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.003838 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.003839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.003840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.003898 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.004162 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.004163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.004164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.004295 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:01.004443 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:01.004662 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 20:48:01.004767 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260926 20:48:01.005546 22650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08: Bootstrap starting.
I20260926 20:48:01.005883 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.997940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:46427 (local address 127.0.0.1:50698)
0926 20:48:00.997978 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:00.997988 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:00.998019 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:00.998157 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:00.998158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:00.998636 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:00.998678 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:00.998679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.999332 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:00.999333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:00.999900 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:00.999902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.000449 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.000450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:01.000534 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:01.000930 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:01.000935 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:01.002206 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:01.004585 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:01.004586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:01.004586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:01.004834 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:01.005154 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:01.005155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:01.005155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:01.005285 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:01.005594 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:01.005595 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:01.005637 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 20:48:01.005717 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 20:48:01.005796 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 20:48:01.005946 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:00.998009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50698 (local address 127.0.98.125:46427)
0926 20:48:00.998054 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:00.998056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:00.998064 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:48:00.998089 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:00.998090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:00.998098 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:00.998142 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:00.998766 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:00.999235 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.000002 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.000280 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.000350 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:01.002297 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:01.002298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:01.002299 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:01.002306 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:01.004475 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.004894 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.004895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.004895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.004949 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.005372 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.005373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.005374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.005517 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:01.005585 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:01.005748 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 20:48:01.005839 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 20:48:01.006238 22800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:48:01.006601 22650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:01.006953 22650 log.cc:824] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08: Log is configured to *not* fsync() on all Append() calls
I20260926 20:48:01.008050 22650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08: No bootstrap required, opened a new log
I20260926 20:48:01.009876 22650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c85d9a522b645dfa65101f943d78e08" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } } peers { permanent_uuid: "e0737988557c41199caac028484016ea" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } } peers { permanent_uuid: "ece21816afef41a1a4ca95782f0c5596" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } }
I20260926 20:48:01.010013 22650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c85d9a522b645dfa65101f943d78e08, State: Initialized, Role: FOLLOWER
I20260926 20:48:01.010103 22650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c85d9a522b645dfa65101f943d78e08" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } } peers { permanent_uuid: "e0737988557c41199caac028484016ea" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } } peers { permanent_uuid: "ece21816afef41a1a4ca95782f0c5596" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } }
Sep 26 20:48:01 dist-test-slave-4hd3 krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260926 20:48:01.010432 22650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:48:01.010439 22807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c85d9a522b645dfa65101f943d78e08" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } } peers { permanent_uuid: "e0737988557c41199caac028484016ea" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } } peers { permanent_uuid: "ece21816afef41a1a4ca95782f0c5596" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } } }
I20260926 20:48:01.010771 22807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 20:48:01 dist-test-slave-4hd3 krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
W20260926 20:48:01.012645 22818 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:48:01.012697 22818 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:48:01.014460 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:01.006489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:37365 (local address 127.0.0.1:40166)
0926 20:48:01.006530 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:01.006539 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:01.006576 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:01.006687 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:01.006687 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:01.006830 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:01.006862 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:01.006864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.007667 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.007668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:01.008290 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:01.008292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.008762 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.008763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:01.008829 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:01.009182 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:01.009187 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:01.011262 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:01.013459 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:01.013460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:01.013460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:01.013658 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:01.013849 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:01.013849 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:01.013850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:01.013901 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:01.014195 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:01.014195 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:01.014227 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 20:48:01.014283 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 20:48:01.014365 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 20:48:01.014897 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:01.005598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50638 (local address 127.0.98.126:33059)
0926 20:48:01.005635 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.005637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.005675 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:48:01.005760 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.005760 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.005769 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.005797 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.006412 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.007050 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.008071 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.008380 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.008437 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:01.011469 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:01.011470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:01.011471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:01.011477 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:01.013545 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.013930 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.013931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.013931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.013986 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.014182 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.014183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.014183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.014295 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:01.014436 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:01.014677 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 20:48:01.014773 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:48:01.015034 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:01.006521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40166 (local address 127.0.98.124:37365)
0926 20:48:01.006554 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.006555 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.006565 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:48:01.006623 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.006624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.006632 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.006682 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.006942 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.007575 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.008379 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.008679 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.008753 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:01.011376 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:01.011377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:01.011377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:01.011384 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:01.013355 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.013734 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.013734 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.013734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.013781 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.013960 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.013961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.013961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.014119 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:01.014175 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:01.014863 (+ 688us) server_negotiation.cc:299] Negotiation successful
0926 20:48:01.014947 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:48:01.013464 22582 master_runner.cc:186] Error getting master registration for 127.0.98.125:46427: OK, instance_id { permanent_uuid: "e0737988557c41199caac028484016ea" instance_seqno: 1790455680820376 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:48:01.016390 22582 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:48:01.017148 22800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596: Bootstrap starting.
I20260926 20:48:01.017747 22800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:01.018044 22800 log.cc:824] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596: Log is configured to *not* fsync() on all Append() calls
I20260926 20:48:01.018733 22800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596: No bootstrap required, opened a new log
I20260926 20:48:01.020533 22800 raft_consensus.cc:348] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c85d9a522b645dfa65101f943d78e08" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } } peers { permanent_uuid: "e0737988557c41199caac028484016ea" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } } peers { permanent_uuid: "ece21816afef41a1a4ca95782f0c5596" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } }
I20260926 20:48:01.020679 22800 raft_consensus.cc:729] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece21816afef41a1a4ca95782f0c5596, State: Initialized, Role: FOLLOWER
I20260926 20:48:01.020776 22800 consensus_queue.cc:260] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c85d9a522b645dfa65101f943d78e08" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } } peers { permanent_uuid: "e0737988557c41199caac028484016ea" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } } peers { permanent_uuid: "ece21816afef41a1a4ca95782f0c5596" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } }
I20260926 20:48:01.021065 22821 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c85d9a522b645dfa65101f943d78e08" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } } peers { permanent_uuid: "e0737988557c41199caac028484016ea" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } } peers { permanent_uuid: "ece21816afef41a1a4ca95782f0c5596" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } } }
I20260926 20:48:01.021148 22800 sys_catalog.cc:564] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:48:01.021219 22821 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596 [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 20:48:01 dist-test-slave-4hd3 krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
W20260926 20:48:01.023258 22832 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:48:01.023314 22832 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:48:01.026322 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:01.016404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50700 (local address 127.0.98.125:46427)
0926 20:48:01.016443 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.016445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.016537 (+ 92us) server_negotiation.cc:407] Connection header received
0926 20:48:01.016601 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.016602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.016613 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.016671 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.017271 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.018019 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.018977 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.019257 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.019345 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:01.022083 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:01.022085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:01.022085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:01.022093 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:01.024709 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.025094 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.025095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.025095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.025151 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.025440 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.025441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.025441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.025577 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:01.025713 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:01.026142 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 20:48:01.026216 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:48:01.024930 22732 master_runner.cc:186] Error getting master registration for 127.0.98.125:46427: OK, instance_id { permanent_uuid: "e0737988557c41199caac028484016ea" instance_seqno: 1790455680820376 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:48:01.027146 22732 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:48:01.027860 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:48:01.028879 22624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5c85d9a522b645dfa65101f943d78e08"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50638
I20260926 20:48:01.029021 22624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 20:48:01.029066 22624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:01.030015 22624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c85d9a522b645dfa65101f943d78e08" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } } peers { permanent_uuid: "e0737988557c41199caac028484016ea" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } } peers { permanent_uuid: "ece21816afef41a1a4ca95782f0c5596" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } }
I20260926 20:48:01.030288 22624 leader_election.cc:288] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [CANDIDATE]: Term 1 election: Requested vote from peers e0737988557c41199caac028484016ea (127.0.98.125:46427), ece21816afef41a1a4ca95782f0c5596 (127.0.98.124:37365)
I20260926 20:48:01.030575 22774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c85d9a522b645dfa65101f943d78e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ece21816afef41a1a4ca95782f0c5596"
I20260926 20:48:01.030712 22774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:01.032039 22774 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c85d9a522b645dfa65101f943d78e08 in term 1.
I20260926 20:48:01.032256 22599 leader_election.cc:302] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c85d9a522b645dfa65101f943d78e08, ece21816afef41a1a4ca95782f0c5596; no voters:
I20260926 20:48:01.032393 22807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:48:01.032577 22807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [term 1 LEADER]: Becoming Leader. State: Replica: 5c85d9a522b645dfa65101f943d78e08, State: Running, Role: LEADER
I20260926 20:48:01.032663 22807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c85d9a522b645dfa65101f943d78e08" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } } peers { permanent_uuid: "e0737988557c41199caac028484016ea" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } } peers { permanent_uuid: "ece21816afef41a1a4ca95782f0c5596" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } }
I20260926 20:48:01.033075 22807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c85d9a522b645dfa65101f943d78e08. Latest consensus state: current_term: 1 leader_uuid: "5c85d9a522b645dfa65101f943d78e08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c85d9a522b645dfa65101f943d78e08" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } } peers { permanent_uuid: "e0737988557c41199caac028484016ea" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } } peers { permanent_uuid: "ece21816afef41a1a4ca95782f0c5596" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } } }
I20260926 20:48:01.033145 22807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [sys.catalog]: This master's current role is: LEADER
I20260926 20:48:01.040771 22836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:48:01.041038 22836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:48:01.046343 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:01.034197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.0.98.125:46427)
0926 20:48:01.034242 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.034244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.034918 (+ 674us) server_negotiation.cc:407] Connection header received
0926 20:48:01.035053 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.035054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.035064 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.035096 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.036321 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.037076 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.038592 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.038909 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.038948 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:01.043496 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:01.043497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:01.043498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:01.043507 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:01.045123 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.045506 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.045507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.045508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.045561 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.045751 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.045752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.045752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.045892 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:01.045961 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:01.046161 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 20:48:01.046245 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:48:01.047492 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:01.033864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.0.98.124:37365)
0926 20:48:01.033920 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.033922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.034584 (+ 662us) server_negotiation.cc:407] Connection header received
0926 20:48:01.034818 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.034819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.034831 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.034865 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.035646 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.036418 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.037501 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.037849 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.037892 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:01.043071 (+ 5179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:01.043072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:01.043073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:01.043080 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:01.045331 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.045898 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.045900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.045900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.045956 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.046239 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.046239 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.046240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.046389 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:01.046457 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:01.047304 (+ 847us) server_negotiation.cc:299] Negotiation successful
0926 20:48:01.047400 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:48:01.052764 22774 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596 [term 1 FOLLOWER]: Refusing update from remote peer 5c85d9a522b645dfa65101f943d78e08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:01.052825 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:01.034438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50652 (local address 127.0.98.126:33059)
0926 20:48:01.034477 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.034478 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.039129 (+ 4651us) server_negotiation.cc:407] Connection header received
0926 20:48:01.039738 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.039739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.039752 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.039784 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.040715 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.041453 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.042643 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.042948 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.043010 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:01.049918 (+ 6908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:01.049920 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:01.049921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:01.049928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:01.051573 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.051972 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.051973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.051973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.052027 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.052217 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.052218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.052218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.052359 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:01.052436 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:01.052646 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 20:48:01.052726 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:48:01.053119 22834 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ece21816afef41a1a4ca95782f0c5596" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:48:01.054512 22821 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c85d9a522b645dfa65101f943d78e08. Latest consensus state: current_term: 1 leader_uuid: "5c85d9a522b645dfa65101f943d78e08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c85d9a522b645dfa65101f943d78e08" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } } peers { permanent_uuid: "e0737988557c41199caac028484016ea" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } } peers { permanent_uuid: "ece21816afef41a1a4ca95782f0c5596" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } } }
I20260926 20:48:01.054605 22821 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:48:01.054653 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } attempt: 3
I20260926 20:48:01.057207 22807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c85d9a522b645dfa65101f943d78e08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c85d9a522b645dfa65101f943d78e08" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } } peers { permanent_uuid: "e0737988557c41199caac028484016ea" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } } peers { permanent_uuid: "ece21816afef41a1a4ca95782f0c5596" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } } }
I20260926 20:48:01.057325 22807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [sys.catalog]: This master's current role is: LEADER
I20260926 20:48:01.058195 22821 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c85d9a522b645dfa65101f943d78e08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c85d9a522b645dfa65101f943d78e08" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } } peers { permanent_uuid: "e0737988557c41199caac028484016ea" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } } peers { permanent_uuid: "ece21816afef41a1a4ca95782f0c5596" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } } }
I20260926 20:48:01.058288 22821 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:48:01.058506 22845 mvcc.cc:205] Tried to move back new op lower bound from 7333706469589426176 to 7333706469510713344. Current Snapshot: MvccSnapshot[applied={T|T < 7333706469589426176}]
I20260926 20:48:01.058764 22844 mvcc.cc:205] Tried to move back new op lower bound from 7333706469589426176 to 7333706469510713344. Current Snapshot: MvccSnapshot[applied={T|T < 7333706469589426176}]
Sep 26 20:48:01 dist-test-slave-4hd3 krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260926 20:48:01.063454 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:01.054908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:37365 (local address 127.0.0.1:40182)
0926 20:48:01.054984 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:01.054993 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:01.055029 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:01.055251 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:01.055252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:01.055403 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:01.055449 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:01.055451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.056388 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.056389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:01.057178 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:01.057180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.057693 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.057694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:01.057751 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:01.058184 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:01.058189 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:01.059628 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:01.062395 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:01.062397 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:01.062397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:01.062606 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:01.062795 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:01.062807 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:01.062807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:01.062874 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:01.063143 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:01.063143 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:01.063201 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 20:48:01.063266 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 20:48:01.063355 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 20:48:01.063786 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:01.055099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.0.98.124:37365)
0926 20:48:01.055133 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.055135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.055141 (+ 6us) server_negotiation.cc:407] Connection header received
0926 20:48:01.055159 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.055159 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.055168 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.055210 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.055527 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.056299 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.057267 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:01.057584 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:01.057642 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:01.059745 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:01.059746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:01.059746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:01.059754 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:01.062301 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.062675 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.062676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.062676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.062745 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:01.062935 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:01.062937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:01.062937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:01.063065 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:01.063134 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:01.063318 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 20:48:01.063394 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:48:01.064311 22836 catalog_manager.cc:1347] Generated new cluster ID: f357a75097a14524ab55d57715262fef
I20260926 20:48:01.064361 22836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:48:01.069087 22725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea: Bootstrap starting.
I20260926 20:48:01.069796 22725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:01.070101 22725 log.cc:824] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea: Log is configured to *not* fsync() on all Append() calls
I20260926 20:48:01.073315 22725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea: No bootstrap required, opened a new log
I20260926 20:48:01.075309 22725 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c85d9a522b645dfa65101f943d78e08" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } } peers { permanent_uuid: "e0737988557c41199caac028484016ea" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } } peers { permanent_uuid: "ece21816afef41a1a4ca95782f0c5596" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } }
I20260926 20:48:01.075448 22725 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0737988557c41199caac028484016ea, State: Initialized, Role: FOLLOWER
I20260926 20:48:01.075527 22725 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c85d9a522b645dfa65101f943d78e08" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } } peers { permanent_uuid: "e0737988557c41199caac028484016ea" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } } peers { permanent_uuid: "ece21816afef41a1a4ca95782f0c5596" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } }
I20260926 20:48:01.076004 22698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:01.076918 22698 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea [term 1 FOLLOWER]: Refusing update from remote peer 5c85d9a522b645dfa65101f943d78e08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:01.077082 22699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c85d9a522b645dfa65101f943d78e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e0737988557c41199caac028484016ea"
I20260926 20:48:01.077247 22836 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:48:01.077893 22699 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c85d9a522b645dfa65101f943d78e08 in term 1.
I20260926 20:48:01.077971 22836 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:48:01.075908 22725 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:48:01.078198 22834 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0737988557c41199caac028484016ea" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:48:01.082459 22836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08: Generated new TSK 0
I20260926 20:48:01.082671 22836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:48:01.075910 22851 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c85d9a522b645dfa65101f943d78e08" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } } peers { permanent_uuid: "e0737988557c41199caac028484016ea" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } } peers { permanent_uuid: "ece21816afef41a1a4ca95782f0c5596" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } } }
I20260926 20:48:01.083098 22851 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:48:01.083184 22851 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c85d9a522b645dfa65101f943d78e08. Latest consensus state: current_term: 1 leader_uuid: "5c85d9a522b645dfa65101f943d78e08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c85d9a522b645dfa65101f943d78e08" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } } peers { permanent_uuid: "e0737988557c41199caac028484016ea" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } } peers { permanent_uuid: "ece21816afef41a1a4ca95782f0c5596" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } } }
I20260926 20:48:01.083276 22851 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:48:01.083294 22866 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:48:01.083359 22866 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:48:01.083454 22851 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c85d9a522b645dfa65101f943d78e08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c85d9a522b645dfa65101f943d78e08" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } } peers { permanent_uuid: "e0737988557c41199caac028484016ea" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } } peers { permanent_uuid: "ece21816afef41a1a4ca95782f0c5596" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } } }
I20260926 20:48:01.083557 22851 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:48:01.084488 22847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c85d9a522b645dfa65101f943d78e08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c85d9a522b645dfa65101f943d78e08" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33059 } } peers { permanent_uuid: "e0737988557c41199caac028484016ea" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46427 } } peers { permanent_uuid: "ece21816afef41a1a4ca95782f0c5596" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 37365 } } }
I20260926 20:48:01.084590 22847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c85d9a522b645dfa65101f943d78e08 [sys.catalog]: This master's current role is: LEADER
I20260926 20:48:01.084908 22858 mvcc.cc:205] Tried to move back new op lower bound from 7333706469687283712 to 7333706469510713344. Current Snapshot: MvccSnapshot[applied={T|T < 7333706469589426176}]
W20260926 20:48:01.109133 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:01.107558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.0.98.125:46427)
0926 20:48:01.107601 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.107604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.108473 (+ 869us) server_negotiation.cc:407] Connection header received
0926 20:48:01.108573 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.108574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.108585 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.108615 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.109056 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50714: BlockingRecv error: recv got EOF from 127.0.0.1:50714 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 20:48:01.110004 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:01.107940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.0.98.124:37365)
0926 20:48:01.107978 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.107980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.109506 (+ 1526us) server_negotiation.cc:407] Connection header received
0926 20:48:01.109601 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.109602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.109612 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.109641 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.109944 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40196: BlockingRecv error: recv got EOF from 127.0.0.1:40196 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 20:48:01.110841 22653 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:01.108230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50656 (local address 127.0.98.126:33059)
0926 20:48:01.108266 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.108267 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.110366 (+ 2099us) server_negotiation.cc:407] Connection header received
0926 20:48:01.110466 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.110467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.110478 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.110506 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.110785 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50656: BlockingRecv error: recv got EOF from 127.0.0.1:50656 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 20:48:01.113261 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:01.112189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50718 (local address 127.0.98.125:46427)
0926 20:48:01.112220 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.112222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.112781 (+ 559us) server_negotiation.cc:407] Connection header received
0926 20:48:01.112886 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.112887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.112903 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.112931 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.113216 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50718: BlockingRecv error: recv got EOF from 127.0.0.1:50718 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 20:48:01.116189 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:01.115429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.0.98.125:46427)
0926 20:48:01.115460 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.115462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.115680 (+ 218us) server_negotiation.cc:407] Connection header received
0926 20:48:01.115814 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.115814 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.115824 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.115854 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.116145 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50734: BlockingRecv error: recv got EOF from 127.0.0.1:50734 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 20:48:01.120007 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:01.119215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50736 (local address 127.0.98.125:46427)
0926 20:48:01.119244 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.119245 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.119528 (+ 283us) server_negotiation.cc:407] Connection header received
0926 20:48:01.119617 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.119618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.119626 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.119654 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.119966 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50736: BlockingRecv error: recv got EOF from 127.0.0.1:50736 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 20:48:01.125759 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:01.125075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.0.98.125:46427)
0926 20:48:01.125107 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.125109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.125313 (+ 204us) server_negotiation.cc:407] Connection header received
0926 20:48:01.125403 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.125404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.125413 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.125440 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.125719 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50752: BlockingRecv error: recv got EOF from 127.0.0.1:50752 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 20:48:01.135867 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:01.134938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50760 (local address 127.0.98.125:46427)
0926 20:48:01.134972 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.134974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.135234 (+ 260us) server_negotiation.cc:407] Connection header received
0926 20:48:01.135350 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.135351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.135363 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.135394 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.135804 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50760: BlockingRecv error: recv got EOF from 127.0.0.1:50760 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 20:48:01.154155 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:01.153214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50764 (local address 127.0.98.125:46427)
0926 20:48:01.153256 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.153259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.153562 (+ 303us) server_negotiation.cc:407] Connection header received
0926 20:48:01.153669 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.153671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.153683 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.153714 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.154094 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50764: BlockingRecv error: recv got EOF from 127.0.0.1:50764 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 20:48:01.188730 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:01.187650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50778 (local address 127.0.98.125:46427)
0926 20:48:01.187692 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.187694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.188046 (+ 352us) server_negotiation.cc:407] Connection header received
0926 20:48:01.188173 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.188174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.188186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.188216 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.188652 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50778: BlockingRecv error: recv got EOF from 127.0.0.1:50778 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 20:48:01.255033 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:01.254279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.0.98.125:46427)
0926 20:48:01.254352 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.254355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.254506 (+ 151us) server_negotiation.cc:407] Connection header received
0926 20:48:01.254550 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.254551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.254561 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.254614 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.254957 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50794: BlockingRecv error: recv got EOF from 127.0.0.1:50794 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 20:48:01.385190 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:01.384400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50802 (local address 127.0.98.125:46427)
0926 20:48:01.384488 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.384491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.384597 (+ 106us) server_negotiation.cc:407] Connection header received
0926 20:48:01.384710 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.384712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.384730 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.384800 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.385139 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50802: BlockingRecv error: recv got EOF from 127.0.0.1:50802 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 20:48:01.643267 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:01.642589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.0.98.125:46427)
0926 20:48:01.642681 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:01.642683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:01.642693 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:48:01.642726 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:01.642727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:01.642738 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:01.642823 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:01.643199 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50810: BlockingRecv error: recv got EOF from 127.0.0.1:50810 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:48:02.023768 22832 catalog_manager.cc:1259] Loaded cluster ID: f357a75097a14524ab55d57715262fef
I20260926 20:48:02.023845 22832 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596: loading cluster ID for follower catalog manager: success
I20260926 20:48:02.024598 22832 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596: acquiring CA information for follower catalog manager: success
I20260926 20:48:02.024950 22832 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ece21816afef41a1a4ca95782f0c5596: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:48:02.083809 22866 catalog_manager.cc:1259] Loaded cluster ID: f357a75097a14524ab55d57715262fef
I20260926 20:48:02.083881 22866 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea: loading cluster ID for follower catalog manager: success
I20260926 20:48:02.084784 22866 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea: acquiring CA information for follower catalog manager: success
I20260926 20:48:02.085098 22866 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e0737988557c41199caac028484016ea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:48:02.159060 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:02.156443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.0.98.125:46427)
0926 20:48:02.156570 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:02.156572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:02.156582 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:48:02.156635 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:02.156636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:02.156687 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:02.156751 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:02.156926 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:02.157460 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:02.158219 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:02.158490 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:02.158545 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:02.158707 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:02.158817 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:02.158884 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:02.158977 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 20:48:02.158995 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260926 20:48:02.163507 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:02.161031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.0.98.124:37365)
0926 20:48:02.161162 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:02.161164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:02.161224 (+ 60us) server_negotiation.cc:407] Connection header received
0926 20:48:02.161283 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:02.161284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:02.161316 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:02.161387 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:02.161485 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:02.161957 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:02.162691 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:02.162880 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:02.162936 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:02.163137 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:02.163239 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:02.163311 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:02.163354 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 20:48:02.163380 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260926 20:48:02.168111 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:02.165390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.0.98.125:46427)
0926 20:48:02.165428 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:02.165430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:02.165440 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:48:02.165499 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:02.165500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:02.165513 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:02.165579 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:02.165669 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:02.166215 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:02.167283 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:02.167570 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:02.167625 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:02.167846 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:02.167882 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:02.167935 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:02.168016 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 20:48:02.168039 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:48:02.168284 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:02.165285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50672 (local address 127.0.98.126:33059)
0926 20:48:02.165522 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:02.165524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:02.165532 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:48:02.165575 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:02.165575 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:02.165627 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:02.165739 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:02.165841 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:02.166360 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:02.167192 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:02.167503 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:02.167568 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:02.167781 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:02.167890 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:02.168128 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:02.168147 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 20:48:02.168177 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":61,"threads_started":1}
I20260926 20:48:02.169003 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:02.165362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.0.98.124:37365)
0926 20:48:02.165503 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:02.165505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:02.165600 (+ 95us) server_negotiation.cc:407] Connection header received
0926 20:48:02.166293 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:02.166294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:02.166302 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:02.166371 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:02.166567 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:02.167278 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:02.168286 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:02.168550 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:02.168590 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:02.168767 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:02.168799 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:02.168850 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:02.168937 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 20:48:02.168957 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260926 20:48:02.170181 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 22582
I20260926 20:48:02.176185 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 22657
I20260926 20:48:02.181661 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 22732
2026-09-26T20:48:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2793 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:48:02 dist-test-slave-4hd3 krb5kdc[22952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:48:02 dist-test-slave-4hd3 krb5kdc[22952](info): set up 2 sockets
Sep 26 20:48:02 dist-test-slave-4hd3 krb5kdc[22952](info): commencing operation
krb5kdc: starting...
W20260926 20:48:03.216054 393 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/test-user.keytab.
Sep 26 20:48:03 dist-test-slave-4hd3 krb5kdc[22952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:48:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:48:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260926 20:48:03.332234 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39917
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39917,127.0.98.125:44161,127.0.98.124:46829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:48:03.413527 22965 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:48:03.413769 22965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:03.413800 22965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:03.413842 22965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:03.414973 22965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:48:03.415032 22965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:03.415066 22965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:03.415100 22965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:48:03.415124 22965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:48:03.416699 22965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39917,127.0.98.125:44161,127.0.98.124:46829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:03.416956 22965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:03.417198 22965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:03.422003 22973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:03.422003 22970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:03.422150 22971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:03.422343 22965 server_base.cc:1034] running on GCE node
I20260926 20:48:03.422544 22965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:03.422814 22965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:03.423954 22965 hybrid_clock.cc:648] HybridClock initialized: now 1790455683423936 us; error 31 us; skew 500 ppm
Sep 26 20:48:03 dist-test-slave-4hd3 krb5kdc[22952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:03.426513 22965 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260926 20:48:03.426982 22965 webserver.cc:460] Webserver started at http://127.0.98.126:37579/ using document root <none> and password file <none>
I20260926 20:48:03.427205 22965 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:03.427273 22965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:03.427423 22965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:03.428534 22965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-0/data/instance:
uuid: "0a1f625b3e554f0a9fa7d3b042fb9866"
format_stamp: "Formatted at 2026-09-26 20:48:03 on dist-test-slave-4hd3"
server_key: "d563df6630546012becb991ec54acfe9"
server_key_iv: "19b7a0db1911b24cfadd190e5edebe5e"
server_key_version: "encryptionkey@0"
I20260926 20:48:03.428889 22965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-0/wal/instance:
uuid: "0a1f625b3e554f0a9fa7d3b042fb9866"
format_stamp: "Formatted at 2026-09-26 20:48:03 on dist-test-slave-4hd3"
server_key: "d563df6630546012becb991ec54acfe9"
server_key_iv: "19b7a0db1911b24cfadd190e5edebe5e"
server_key_version: "encryptionkey@0"
I20260926 20:48:03.430168 22965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:48:03.430871 22980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:03.431041 22965 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:48:03.431169 22965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-0/wal
uuid: "0a1f625b3e554f0a9fa7d3b042fb9866"
format_stamp: "Formatted at 2026-09-26 20:48:03 on dist-test-slave-4hd3"
server_key: "d563df6630546012becb991ec54acfe9"
server_key_iv: "19b7a0db1911b24cfadd190e5edebe5e"
server_key_version: "encryptionkey@0"
I20260926 20:48:03.431241 22965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:03.438795 22965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:03.441112 22965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:03.441278 22965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:03.445377 22965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:39917
I20260926 20:48:03.445410 23032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:39917 every 8 connection(s)
I20260926 20:48:03.445740 22965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-0/data/info.pb
I20260926 20:48:03.446645 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 22965
I20260926 20:48:03.446720 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-0/wal/instance
I20260926 20:48:03.446686 23033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:03.446930 393 external_mini_cluster.cc:1428] Setting key ff49f54c1a7e4a3894e1b334ef60e5c3
I20260926 20:48:03.448304 23033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.125@KRBTEST.COM; defaulting to no policy
Sep 26 20:48:03 dist-test-slave-4hd3 krb5kdc[22952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.125@KRBTEST.COM" created.
I20260926 20:48:03.457751 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.449048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33408 (local address 127.0.98.126:39917)
0926 20:48:03.449233 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.449237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.449337 (+ 100us) server_negotiation.cc:407] Connection header received
0926 20:48:03.449887 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.449889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.449914 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.450012 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.450523 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.451311 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.452261 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.452563 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.452688 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.454677 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:03.454683 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:03.454683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:03.454690 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:03.456148 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.456581 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.456583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.456584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.456660 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.456883 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.456884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.456885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.457094 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:03.457187 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.457408 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.457581 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":161,"thread_start_us":93,"threads_started":1}
I20260926 20:48:03.457752 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.449010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39917 (local address 127.0.0.1:33408)
0926 20:48:03.449179 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.449224 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:03.449731 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:03.450025 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:03.450028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:03.450281 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:03.450423 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:03.450427 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.451427 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.451428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:03.452167 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:03.452169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.452675 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.452676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:03.452738 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.453266 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:03.453274 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:03.454592 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:03.456241 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:03.456247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:03.456249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:03.456488 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:03.456729 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:03.456730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:03.456730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:03.456820 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:03.457167 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:03.457171 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:03.457238 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 20:48:03.457328 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:48:03.457513 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260926 20:48:03.458423 23033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:48:03.458760 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.458645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:44161 (local address 127.0.0.1:39584)
0926 20:48:03.458690 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.458731 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:44161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 20:48:03.458881 22982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.125:44161: connect: Connection refused (error 111)
I20260926 20:48:03.459287 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.459197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:44161 (local address 127.0.0.1:39592)
0926 20:48:03.459243 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.459266 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:44161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 20:48:03.459391 23033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:44161: Network error: Client connection negotiation failed: client connection to 127.0.98.125:44161: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/kudu_127.0.98.125.keytab.
Entry for principal kudu/127.0.98.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/kudu_127.0.98.125.keytab.
I20260926 20:48:03.471892 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:44161
--webserver_interface=127.0.98.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/kudu_127.0.98.125.keytab
--principal=kudu/127.0.98.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39917,127.0.98.125:44161,127.0.98.124:46829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:48:03.508669 23033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } attempt: 1
I20260926 20:48:03.509229 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.509081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:44161 (local address 127.0.0.1:39602)
0926 20:48:03.509155 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.509200 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:44161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 20:48:03.509622 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.509537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:44161 (local address 127.0.0.1:39618)
0926 20:48:03.509578 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.509604 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:44161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 20:48:03.509728 23033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:44161: Network error: Client connection negotiation failed: client connection to 127.0.98.125:44161: connect: Connection refused (error 111)
W20260926 20:48:03.552699 23040 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:48:03.552901 23040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:03.552920 23040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:03.552940 23040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:03.554057 23040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:48:03.554098 23040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:03.554112 23040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:03.554126 23040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:48:03.554180 23040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:48:03.555641 23040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39917,127.0.98.125:44161,127.0.98.124:46829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/kudu_127.0.98.125.keytab
--principal=kudu/127.0.98.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:44161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:03.555923 23040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:03.556145 23040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:03.560985 23046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:03.560992 23045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:03.561019 23040 server_base.cc:1034] running on GCE node
W20260926 20:48:03.560992 23048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:03.561432 23040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:03.561687 23040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:03.562852 23040 hybrid_clock.cc:648] HybridClock initialized: now 1790455683562831 us; error 41 us; skew 500 ppm
Sep 26 20:48:03 dist-test-slave-4hd3 krb5kdc[22952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:03.565287 23040 init.cc:377] Logged in from keytab as kudu/127.0.98.125@KRBTEST.COM (short username kudu)
I20260926 20:48:03.565665 23040 webserver.cc:460] Webserver started at http://127.0.98.125:39941/ using document root <none> and password file <none>
I20260926 20:48:03.565841 23040 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:03.565881 23040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:03.565968 23040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:03.567027 23040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-1/data/instance:
uuid: "f103ca7781494d03a60ccf18f7e79de8"
format_stamp: "Formatted at 2026-09-26 20:48:03 on dist-test-slave-4hd3"
server_key: "5e0366921e8ee54023bf0dfe25b0bbf2"
server_key_iv: "a6f54f7a018a8aa344fda73a3074511e"
server_key_version: "encryptionkey@0"
I20260926 20:48:03.567384 23040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-1/wal/instance:
uuid: "f103ca7781494d03a60ccf18f7e79de8"
format_stamp: "Formatted at 2026-09-26 20:48:03 on dist-test-slave-4hd3"
server_key: "5e0366921e8ee54023bf0dfe25b0bbf2"
server_key_iv: "a6f54f7a018a8aa344fda73a3074511e"
server_key_version: "encryptionkey@0"
I20260926 20:48:03.568760 23040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:48:03.569567 23055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:03.569782 23040 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:48:03.569865 23040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-1/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-1/wal
uuid: "f103ca7781494d03a60ccf18f7e79de8"
format_stamp: "Formatted at 2026-09-26 20:48:03 on dist-test-slave-4hd3"
server_key: "5e0366921e8ee54023bf0dfe25b0bbf2"
server_key_iv: "a6f54f7a018a8aa344fda73a3074511e"
server_key_version: "encryptionkey@0"
I20260926 20:48:03.569919 23040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:03.577888 23033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } attempt: 2
I20260926 20:48:03.578377 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.578250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:44161 (local address 127.0.0.1:39634)
0926 20:48:03.578310 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.578349 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:44161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 20:48:03.578737 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.578663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:44161 (local address 127.0.0.1:39642)
0926 20:48:03.578700 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.578723 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:44161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 20:48:03.578840 23033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:44161: Network error: Client connection negotiation failed: client connection to 127.0.98.125:44161: connect: Connection refused (error 111)
I20260926 20:48:03.582123 23040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:03.584390 23040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:03.584498 23040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:03.588279 23040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.125:44161
I20260926 20:48:03.588311 23107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.125:44161 every 8 connection(s)
I20260926 20:48:03.588682 23040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-1/data/info.pb
I20260926 20:48:03.589756 23108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:03.591303 23108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:48:03.596582 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 23040
I20260926 20:48:03.596681 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-1/wal/instance
I20260926 20:48:03.596930 393 external_mini_cluster.cc:1428] Setting key 74294cb834a4cf6a099527d40f9a91d8
Sep 26 20:48:03 dist-test-slave-4hd3 krb5kdc[22952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:48:03.601215 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.592043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33416 (local address 127.0.98.126:39917)
0926 20:48:03.592141 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.592143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.592349 (+ 206us) server_negotiation.cc:407] Connection header received
0926 20:48:03.592935 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.592936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.592947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.593010 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.593631 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.594114 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.595090 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.595283 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.595344 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.598267 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:03.598268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:03.598268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:03.598277 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:03.599840 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.600246 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.600247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.600248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.600307 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.600570 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.600571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.600571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.600687 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:03.600754 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.601035 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.601120 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:48:03.601426 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.592014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39917 (local address 127.0.0.1:33416)
0926 20:48:03.592190 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.592238 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:03.592817 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:03.593088 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:03.593092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:03.593341 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:03.593529 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:03.593533 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.594211 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.594212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:03.594982 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:03.594984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.595396 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.595397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:03.595518 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.596205 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:03.596215 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:03.598150 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:03.599923 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:03.599930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:03.599932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:03.600166 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:03.600372 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:03.600373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:03.600373 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:03.600489 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:03.600757 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:03.600764 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:03.600908 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 20:48:03.601015 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 20:48:03.601185 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
I20260926 20:48:03.601987 23108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.124@KRBTEST.COM" created.
Sep 26 20:48:03 dist-test-slave-4hd3 krb5kdc[22952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260926 20:48:03.610715 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.602347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:44161 (local address 127.0.0.1:39646)
0926 20:48:03.603048 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.603066 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:03.603136 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:03.603284 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:03.603285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:03.603436 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:03.603488 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:03.603490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.604395 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.604396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:03.604900 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:03.604901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.605400 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.605401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:03.605465 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.605816 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:03.605820 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:03.607509 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:03.609688 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:03.609688 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:03.609689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:03.609842 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:03.610073 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:03.610073 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:03.610074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:03.610126 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:03.610404 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:03.610405 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:03.610441 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 20:48:03.610501 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 20:48:03.610606 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688}
I20260926 20:48:03.610800 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.602414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39646 (local address 127.0.98.125:44161)
0926 20:48:03.602639 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.602644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.603144 (+ 500us) server_negotiation.cc:407] Connection header received
0926 20:48:03.603189 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.603192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.603205 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.603277 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.603575 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.604301 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.604991 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.605330 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.605403 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.607610 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:03.607618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:03.607618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:03.607625 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:03.609594 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.609930 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.609933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.609933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.609987 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.610190 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.610191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.610191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.610336 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:03.610426 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.610538 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.610674 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":173,"thread_start_us":95,"threads_started":1}
I20260926 20:48:03.611351 23108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:48:03.611768 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.611594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:46829 (local address 127.0.0.1:57186)
0926 20:48:03.611649 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.611707 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:46829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 20:48:03.611917 23057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.124:46829: connect: Connection refused (error 111)
I20260926 20:48:03.612362 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.612250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:46829 (local address 127.0.0.1:57198)
0926 20:48:03.612303 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.612333 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:46829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 20:48:03.612488 23108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:46829: Network error: Client connection negotiation failed: client connection to 127.0.98.124:46829: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/kudu_127.0.98.124.keytab.
Entry for principal kudu/127.0.98.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/kudu_127.0.98.124.keytab.
I20260926 20:48:03.622669 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDRmUyo/build/release/bin/kudu
/tmp/dist-test-taskDRmUyo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:46829
--webserver_interface=127.0.98.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/kudu_127.0.98.124.keytab
--principal=kudu/127.0.98.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39917,127.0.98.125:44161,127.0.98.124:46829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:48:03.661727 23108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } attempt: 1
I20260926 20:48:03.662288 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.662137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:46829 (local address 127.0.0.1:57214)
0926 20:48:03.662204 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.662252 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:46829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 20:48:03.662680 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.662583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:46829 (local address 127.0.0.1:57224)
0926 20:48:03.662630 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.662657 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:46829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 20:48:03.662815 23108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:46829: Network error: Client connection negotiation failed: client connection to 127.0.98.124:46829: connect: Connection refused (error 111)
I20260926 20:48:03.670035 23033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } attempt: 3
Sep 26 20:48:03 dist-test-slave-4hd3 krb5kdc[22952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260926 20:48:03.677057 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.670366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:44161 (local address 127.0.0.1:39650)
0926 20:48:03.670427 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.670436 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:03.670501 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:03.670660 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:03.670661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:03.670825 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:03.670887 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:03.670889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.671559 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.671561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:03.672171 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:03.672173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.672589 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.672590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:03.672673 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.673143 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:03.673155 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:03.674336 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:03.676015 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:03.676016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:03.676017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:03.676186 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:03.676393 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:03.676394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:03.676394 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:03.676455 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:03.676704 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:03.676705 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:03.676750 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 20:48:03.676839 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:48:03.676939 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 20:48:03.677050 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.670422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39650 (local address 127.0.98.125:44161)
0926 20:48:03.670483 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.670485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.670504 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:48:03.670568 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.670569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.670584 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.670655 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.670968 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.671458 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.672278 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.672515 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.672595 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.674422 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:03.674424 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:03.674424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:03.674433 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:03.675918 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.676259 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.676260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.676261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.676319 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.676519 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.676520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.676520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.676634 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:03.676712 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.676863 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.676939 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:48:03.677420 23033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:48:03.677758 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.677659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:46829 (local address 127.0.0.1:57226)
0926 20:48:03.677708 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.677735 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:46829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:48:03.678136 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.678049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:46829 (local address 127.0.0.1:57234)
0926 20:48:03.678100 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.678122 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:46829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 20:48:03.678242 23033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:46829: Network error: Client connection negotiation failed: client connection to 127.0.98.124:46829: connect: Connection refused (error 111)
W20260926 20:48:03.705094 23115 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:48:03.705304 23115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:03.705322 23115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:03.705341 23115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:03.706395 23115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:48:03.706429 23115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:03.706442 23115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:03.706454 23115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:48:03.706466 23115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:48:03.707953 23115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39917,127.0.98.125:44161,127.0.98.124:46829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/krb5kdc/kudu_127.0.98.124.keytab
--principal=kudu/127.0.98.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:46829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:03.708150 23115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:03.708318 23115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:48:03.709367 23033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } attempt: 1
I20260926 20:48:03.709829 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.709701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:46829 (local address 127.0.0.1:57250)
0926 20:48:03.709765 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.709803 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:46829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 20:48:03.710273 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.710163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:46829 (local address 127.0.0.1:57256)
0926 20:48:03.710226 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.710253 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:46829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 20:48:03.710390 23033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:46829: Network error: Client connection negotiation failed: client connection to 127.0.98.124:46829: connect: Connection refused (error 111)
W20260926 20:48:03.713312 23120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:03.713326 23121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:03.713326 23123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:03.713500 23115 server_base.cc:1034] running on GCE node
I20260926 20:48:03.713790 23115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:03.714082 23115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:03.715257 23115 hybrid_clock.cc:648] HybridClock initialized: now 1790455683715232 us; error 61 us; skew 500 ppm
Sep 26 20:48:03 dist-test-slave-4hd3 krb5kdc[22952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:03.717877 23115 init.cc:377] Logged in from keytab as kudu/127.0.98.124@KRBTEST.COM (short username kudu)
I20260926 20:48:03.718350 23115 webserver.cc:460] Webserver started at http://127.0.98.124:43635/ using document root <none> and password file <none>
I20260926 20:48:03.718581 23115 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:03.718649 23115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:03.718801 23115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:03.719955 23115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-2/data/instance:
uuid: "6bea3ddab804463393a0569a096e29c3"
format_stamp: "Formatted at 2026-09-26 20:48:03 on dist-test-slave-4hd3"
server_key: "a5f33174fbdbcfb966162477479c1a58"
server_key_iv: "948ab155c116d805ac67b5ba47fd52ad"
server_key_version: "encryptionkey@0"
I20260926 20:48:03.720325 23115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-2/wal/instance:
uuid: "6bea3ddab804463393a0569a096e29c3"
format_stamp: "Formatted at 2026-09-26 20:48:03 on dist-test-slave-4hd3"
server_key: "a5f33174fbdbcfb966162477479c1a58"
server_key_iv: "948ab155c116d805ac67b5ba47fd52ad"
server_key_version: "encryptionkey@0"
I20260926 20:48:03.721663 23115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:48:03.722462 23130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:03.722635 23115 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:48:03.722740 23115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-2/data,/tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-2/wal
uuid: "6bea3ddab804463393a0569a096e29c3"
format_stamp: "Formatted at 2026-09-26 20:48:03 on dist-test-slave-4hd3"
server_key: "a5f33174fbdbcfb966162477479c1a58"
server_key_iv: "948ab155c116d805ac67b5ba47fd52ad"
server_key_version: "encryptionkey@0"
I20260926 20:48:03.722813 23115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:03.730969 23108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } attempt: 2
I20260926 20:48:03.731405 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.731275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:46829 (local address 127.0.0.1:57266)
0926 20:48:03.731337 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.731375 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:46829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 20:48:03.731827 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.731684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:46829 (local address 127.0.0.1:57268)
0926 20:48:03.731777 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.731811 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:46829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260926 20:48:03.731930 23108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:46829: Network error: Client connection negotiation failed: client connection to 127.0.98.124:46829: connect: Connection refused (error 111)
I20260926 20:48:03.740950 23115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:03.742754 23115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:03.742870 23115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:03.746978 23115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.124:46829
I20260926 20:48:03.747020 23182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.124:46829 every 8 connection(s)
I20260926 20:48:03.747408 23115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-2/data/info.pb
I20260926 20:48:03.748477 23183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:03.750066 23183 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:48:03 dist-test-slave-4hd3 krb5kdc[22952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:48:03.757543 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDRmUyo/build/release/bin/kudu as pid 23115
I20260926 20:48:03.757673 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDRmUyo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455416624047-393-0/minicluster-data/master-2/wal/instance
I20260926 20:48:03.757933 393 external_mini_cluster.cc:1428] Setting key 8fd91b5ed1f1e5934c3c0e5d6db63072
I20260926 20:48:03.759644 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.750848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33418 (local address 127.0.98.126:39917)
0926 20:48:03.750966 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.750969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.751170 (+ 201us) server_negotiation.cc:407] Connection header received
0926 20:48:03.751734 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.751736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.751750 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.751830 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.752476 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.752947 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.753968 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.754166 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.754274 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.756436 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:03.756438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:03.756438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:03.756446 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:03.758095 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.758613 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.758614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.758614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.758699 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.758943 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.758944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.758946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.759112 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:03.759192 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.759517 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.759582 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:48:03.759902 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.750814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39917 (local address 127.0.0.1:33418)
0926 20:48:03.751026 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.751076 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:03.751589 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:03.751911 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:03.751915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:03.752179 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:03.752370 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:03.752374 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.753061 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.753064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:03.753865 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:03.753867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.754297 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.754299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:03.754415 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.754945 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:03.754953 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:03.756339 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:03.758191 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:03.758198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:03.758200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:03.758540 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:03.758778 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:03.758778 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:03.758778 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:03.758875 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:03.759210 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:03.759216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:03.759338 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 20:48:03.759492 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 20:48:03.759659 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":94,"threads_started":1}
I20260926 20:48:03.760581 23183 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:48:03 dist-test-slave-4hd3 krb5kdc[22952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
Sep 26 20:48:03 dist-test-slave-4hd3 krb5kdc[22952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260926 20:48:03.767379 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.758329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57270 (local address 127.0.98.124:46829)
0926 20:48:03.758484 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.758488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.758508 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:48:03.758576 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.758579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.758595 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.758664 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.759438 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.760490 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.761385 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.761708 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.761800 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.764372 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:03.764379 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:03.764380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:03.764387 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:03.766017 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.766399 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.766402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.766403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.766486 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.766673 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.766674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.766675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.766860 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:03.767009 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.767142 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.767249 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260926 20:48:03.767509 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.760826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:44161 (local address 127.0.0.1:39660)
0926 20:48:03.760873 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.760883 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:03.760919 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:03.761128 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:03.761129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:03.761267 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:03.761315 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:03.761316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.761943 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.761944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:03.762451 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:03.762452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.762864 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.762865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:03.762934 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.763318 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:03.763324 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:03.764646 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:03.766385 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:03.766385 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:03.766386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:03.766616 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:03.766818 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:03.766819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:03.766819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:03.766874 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:03.767212 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:03.767213 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:03.767251 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 20:48:03.767325 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:48:03.767403 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 20:48:03.767549 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.760908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39660 (local address 127.0.98.125:44161)
0926 20:48:03.760976 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.760978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.760986 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:48:03.761051 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.761052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.761060 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.761126 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.761400 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.761857 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.762560 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.762778 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.762867 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.764715 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:03.764716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:03.764716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:03.764724 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:03.766287 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.766702 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.766703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.766703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.766758 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.766940 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.766941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.766941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.767117 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:03.767179 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.767356 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.767436 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:48:03.768327 23183 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:48:03 dist-test-slave-4hd3 krb5kdc[22952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
Sep 26 20:48:03 dist-test-slave-4hd3 krb5kdc[22952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260926 20:48:03.775810 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.768600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:46829 (local address 127.0.0.1:57278)
0926 20:48:03.768678 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.768686 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:03.768753 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:03.768860 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:03.768861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:03.769018 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:03.769072 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:03.769074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.769912 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.769912 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:03.770703 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:03.770706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.771151 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.771151 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:03.771190 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.771562 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:03.771566 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:03.772915 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:03.774795 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:03.774796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:03.774796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:03.774947 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:03.775149 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:03.775149 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:03.775149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:03.775195 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:03.775448 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:03.775449 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:03.775495 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 20:48:03.775574 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:48:03.775665 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 20:48:03.775887 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.768643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.0.98.124:46829)
0926 20:48:03.768676 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.768678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.768719 (+ 41us) server_negotiation.cc:407] Connection header received
0926 20:48:03.768800 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.768800 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.768809 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.768855 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.769154 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.769832 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.770785 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.771047 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.771152 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.773005 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:03.773006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:03.773007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:03.773014 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:03.774719 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.775021 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.775022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.775022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.775080 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.775252 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.775252 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.775253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.775378 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:03.775433 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.775605 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.775753 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":18}
I20260926 20:48:03.776544 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.768549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33430 (local address 127.0.98.126:39917)
0926 20:48:03.768647 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.768649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.768657 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:48:03.768681 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.768682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.768693 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.768749 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.769318 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.769991 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.770909 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.771191 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.771248 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.773740 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:03.773742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:03.773742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:03.773750 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:03.775249 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.775600 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.775601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.775602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.775670 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.775934 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.775936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.775936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.776059 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:03.776156 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.776365 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.776442 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 20:48:03.777557 23183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3: Bootstrap starting.
I20260926 20:48:03.778550 23183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:03.778858 23183 log.cc:824] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:48:03.779786 23183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3: No bootstrap required, opened a new log
Sep 26 20:48:03 dist-test-slave-4hd3 krb5kdc[22952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260926 20:48:03.781608 23183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } } peers { permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } } peers { permanent_uuid: "6bea3ddab804463393a0569a096e29c3" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } }
I20260926 20:48:03.781780 23183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bea3ddab804463393a0569a096e29c3, State: Initialized, Role: FOLLOWER
I20260926 20:48:03.781876 23183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } } peers { permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } } peers { permanent_uuid: "6bea3ddab804463393a0569a096e29c3" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } }
I20260926 20:48:03.782155 23189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } } peers { permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } } peers { permanent_uuid: "6bea3ddab804463393a0569a096e29c3" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } } }
I20260926 20:48:03.782204 23183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:48:03.782262 23189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:48:03.784185 23200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:48:03.784260 23200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:48:03.785168 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.777129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.0.98.125:44161)
0926 20:48:03.777168 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.777170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.777203 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:48:03.777262 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.777263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.777274 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.777336 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.777826 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.778555 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.779397 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.779681 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.779782 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.782165 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:03.782166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:03.782167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:03.782174 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:03.783919 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.784345 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.784346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.784346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.784405 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.784631 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.784632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.784633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.784776 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:03.784935 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.785001 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.785078 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:48:03.785564 23033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } attempt: 2
I20260926 20:48:03.784493 23115 master_runner.cc:186] Error getting master registration for 127.0.98.126:39917: OK, instance_id { permanent_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" instance_seqno: 1790455683444239 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:48:03.786512 23115 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:48:03.787550 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:48:03.788561 23007 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33430
Sep 26 20:48:03 dist-test-slave-4hd3 krb5kdc[22952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260926 20:48:03.794196 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.786653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:46829 (local address 127.0.0.1:57288)
0926 20:48:03.786725 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.786736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:03.786785 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:03.786920 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:03.786921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:03.787059 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:03.787096 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:03.787097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.788157 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.788158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:03.788927 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:03.788930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.789391 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.789392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:03.789445 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.789873 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:03.789877 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:03.791268 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:03.793095 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:03.793096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:03.793096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:03.793311 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:03.793518 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:03.793519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:03.793519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:03.793580 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:03.793881 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:03.793882 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:03.793925 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 20:48:03.794004 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:48:03.794093 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 20:48:03.794198 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.786694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57288 (local address 127.0.98.124:46829)
0926 20:48:03.786729 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.786731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.786770 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:48:03.786832 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.786833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.786842 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.786902 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.787169 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.788055 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.789019 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.789305 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.789377 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.791351 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:03.791352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:03.791352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:03.791359 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:03.793004 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.793388 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.793389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.793390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.793449 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.793646 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.793647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.793647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.793809 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:03.793881 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.794037 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.794112 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:48:03.796356 23033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866: Bootstrap starting.
I20260926 20:48:03.797092 23033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:03.797389 23033 log.cc:824] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866: Log is configured to *not* fsync() on all Append() calls
I20260926 20:48:03.798367 23033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866: No bootstrap required, opened a new log
I20260926 20:48:03.799705 23033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } } peers { permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } } peers { permanent_uuid: "6bea3ddab804463393a0569a096e29c3" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } }
I20260926 20:48:03.799883 23033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a1f625b3e554f0a9fa7d3b042fb9866, State: Initialized, Role: FOLLOWER
I20260926 20:48:03.799954 23033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } } peers { permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } } peers { permanent_uuid: "6bea3ddab804463393a0569a096e29c3" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } }
I20260926 20:48:03.800244 23033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:48:03.800266 23007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 20:48:03.800208 23203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } } peers { permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } } peers { permanent_uuid: "6bea3ddab804463393a0569a096e29c3" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } } }
I20260926 20:48:03.800388 23203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:48:03.800325 23007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:03.801230 23007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } } peers { permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } } peers { permanent_uuid: "6bea3ddab804463393a0569a096e29c3" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } }
I20260926 20:48:03.801776 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6bea3ddab804463393a0569a096e29c3"
I20260926 20:48:03.801900 23157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:03.802702 23157 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a1f625b3e554f0a9fa7d3b042fb9866 in term 1.
I20260926 20:48:03.802970 22982 leader_election.cc:302] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a1f625b3e554f0a9fa7d3b042fb9866, 6bea3ddab804463393a0569a096e29c3; no voters:
I20260926 20:48:03.801479 23007 leader_election.cc:288] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [CANDIDATE]: Term 1 election: Requested vote from peers f103ca7781494d03a60ccf18f7e79de8 (127.0.98.125:44161), 6bea3ddab804463393a0569a096e29c3 (127.0.98.124:46829)
I20260926 20:48:03.803151 23203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:48:03.803552 23203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [term 1 LEADER]: Becoming Leader. State: Replica: 0a1f625b3e554f0a9fa7d3b042fb9866, State: Running, Role: LEADER
I20260926 20:48:03.803663 23203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } } peers { permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } } peers { permanent_uuid: "6bea3ddab804463393a0569a096e29c3" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } }
I20260926 20:48:03.804342 23203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a1f625b3e554f0a9fa7d3b042fb9866. Latest consensus state: current_term: 1 leader_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } } peers { permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } } peers { permanent_uuid: "6bea3ddab804463393a0569a096e29c3" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } } }
I20260926 20:48:03.804431 23203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [sys.catalog]: This master's current role is: LEADER
W20260926 20:48:03.802203 23214 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:48:03.804960 23214 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:48:03.805397 22965 master_runner.cc:186] Error getting master registration for 127.0.98.125:44161: OK, instance_id { permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" instance_seqno: 1790455683587122 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:48:03.807783 22965 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:48:03.808337 23223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:48:03.808532 23223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:48:03.812620 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.804082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33438 (local address 127.0.98.126:39917)
0926 20:48:03.804119 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.804121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.805236 (+ 1115us) server_negotiation.cc:407] Connection header received
0926 20:48:03.805341 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.805342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.805353 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.805382 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.806210 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.806914 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.807930 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.808133 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.808213 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.809669 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:03.809671 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:03.809686 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:03.809694 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:03.811317 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.811735 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.811736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.811736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.811787 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.812021 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.812022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.812022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.812153 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:03.812236 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.812439 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.812509 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 20:48:03.817463 23157 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3 [term 1 FOLLOWER]: Refusing update from remote peer 0a1f625b3e554f0a9fa7d3b042fb9866: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:03.818308 23216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bea3ddab804463393a0569a096e29c3" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:48:03.820115 23189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a1f625b3e554f0a9fa7d3b042fb9866. Latest consensus state: current_term: 1 leader_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } } peers { permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } } peers { permanent_uuid: "6bea3ddab804463393a0569a096e29c3" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } } }
I20260926 20:48:03.820214 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.804770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57300 (local address 127.0.98.124:46829)
0926 20:48:03.804814 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.804816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.812584 (+ 7768us) server_negotiation.cc:407] Connection header received
0926 20:48:03.812674 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.812675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.812686 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.812718 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.813381 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.814072 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.815071 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.815359 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.815392 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.816810 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:03.816811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:03.816812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:03.816818 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:03.818719 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.819183 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.819184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.819184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.819239 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.819480 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.819481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.819481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.819623 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:03.819693 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.819963 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.820044 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 20:48:03.820226 23189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:48:03.823069 23108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } attempt: 3
I20260926 20:48:03.824141 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.804460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.0.98.125:44161)
0926 20:48:03.804501 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.804503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.811894 (+ 7391us) server_negotiation.cc:407] Connection header received
0926 20:48:03.812043 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.812045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.812058 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.812112 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.812736 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.813445 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.814419 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.815172 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.815240 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.820556 (+ 5316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:03.820570 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:03.820571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:03.820579 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:03.822267 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.822644 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.822645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.822645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.822695 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.822923 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.822924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.822925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.823053 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:03.823133 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.823909 (+ 776us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.824051 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:48:03.826808 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } } peers { permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } } peers { permanent_uuid: "6bea3ddab804463393a0569a096e29c3" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } } }
I20260926 20:48:03.826903 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [sys.catalog]: This master's current role is: LEADER
I20260926 20:48:03.827319 23189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } } peers { permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } } peers { permanent_uuid: "6bea3ddab804463393a0569a096e29c3" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } } }
I20260926 20:48:03.827425 23189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:48:03.827459 23227 mvcc.cc:205] Tried to move back new op lower bound from 7333706480913227776 to 7333706480861810688. Current Snapshot: MvccSnapshot[applied={T|T < 7333706480913227776}]
Sep 26 20:48:03 dist-test-slave-4hd3 krb5kdc[22952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260926 20:48:03.827625 23226 mvcc.cc:205] Tried to move back new op lower bound from 7333706480913227776 to 7333706480861810688. Current Snapshot: MvccSnapshot[applied={T|T < 7333706480913227776}]
I20260926 20:48:03.830617 23223 catalog_manager.cc:1347] Generated new cluster ID: 0815af7ea0cb470e99b59c879e0de8e2
I20260926 20:48:03.830670 23223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:48:03.831113 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.823289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:46829 (local address 127.0.0.1:57302)
0926 20:48:03.823332 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:03.823343 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:03.823379 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:03.823756 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:03.823757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:03.823920 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:03.823958 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:03.823960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.824853 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.824854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:03.825653 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:03.825655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.826107 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.826108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:03.826173 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.826543 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:03.826547 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:03.827985 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:03.829819 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:03.829820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:03.829820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:03.830033 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:03.830451 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:03.830452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:03.830452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:03.830517 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:03.830813 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:03.830814 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:03.830857 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 20:48:03.830919 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 20:48:03.831010 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 20:48:03.831449 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.823583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57302 (local address 127.0.98.124:46829)
0926 20:48:03.823618 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.823620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.823641 (+ 21us) server_negotiation.cc:407] Connection header received
0926 20:48:03.823659 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.823659 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.823668 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.824037 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.824079 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.824769 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.825737 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.826046 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.826104 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.828073 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:03.828074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:03.828075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:03.828081 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:03.829750 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.830258 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.830259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.830260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.830381 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:03.830587 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:03.830588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:03.830588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:03.830727 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:03.830787 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.831289 (+ 502us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.831357 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 20:48:03.832942 23108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8: Bootstrap starting.
I20260926 20:48:03.833688 23108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:03.834039 23108 log.cc:824] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8: Log is configured to *not* fsync() on all Append() calls
I20260926 20:48:03.839332 23223 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:48:03.840483 23223 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:48:03.848493 23108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8: No bootstrap required, opened a new log
I20260926 20:48:03.851496 23108 raft_consensus.cc:348] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } } peers { permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } } peers { permanent_uuid: "6bea3ddab804463393a0569a096e29c3" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } }
I20260926 20:48:03.851789 23108 raft_consensus.cc:729] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f103ca7781494d03a60ccf18f7e79de8, State: Initialized, Role: FOLLOWER
I20260926 20:48:03.851995 23108 consensus_queue.cc:260] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } } peers { permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } } peers { permanent_uuid: "6bea3ddab804463393a0569a096e29c3" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } }
I20260926 20:48:03.852669 23082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f103ca7781494d03a60ccf18f7e79de8"
I20260926 20:48:03.852698 23081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:03.853783 23081 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8 [term 1 FOLLOWER]: Refusing update from remote peer 0a1f625b3e554f0a9fa7d3b042fb9866: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:03.854094 23216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:48:03.854843 23082 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a1f625b3e554f0a9fa7d3b042fb9866 in term 1.
I20260926 20:48:03.852624 23108 sys_catalog.cc:564] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:48:03.859807 23236 sys_catalog.cc:455] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } } peers { permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } } peers { permanent_uuid: "6bea3ddab804463393a0569a096e29c3" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } } }
I20260926 20:48:03.859900 23236 sys_catalog.cc:458] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:48:03.859944 23236 sys_catalog.cc:455] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a1f625b3e554f0a9fa7d3b042fb9866. Latest consensus state: current_term: 1 leader_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } } peers { permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } } peers { permanent_uuid: "6bea3ddab804463393a0569a096e29c3" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } } }
I20260926 20:48:03.859992 23236 sys_catalog.cc:458] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:48:03.860019 23236 sys_catalog.cc:455] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } } peers { permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } } peers { permanent_uuid: "6bea3ddab804463393a0569a096e29c3" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } } }
I20260926 20:48:03.860056 23236 sys_catalog.cc:458] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:48:03.861236 23234 mvcc.cc:205] Tried to move back new op lower bound from 7333706481000157184 to 7333706480861810688. Current Snapshot: MvccSnapshot[applied={T|T < 7333706480913227776}]
I20260926 20:48:03.865959 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1f625b3e554f0a9fa7d3b042fb9866" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39917 } } peers { permanent_uuid: "f103ca7781494d03a60ccf18f7e79de8" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 44161 } } peers { permanent_uuid: "6bea3ddab804463393a0569a096e29c3" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 46829 } } }
I20260926 20:48:03.866058 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866 [sys.catalog]: This master's current role is: LEADER
I20260926 20:48:03.867982 23223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a1f625b3e554f0a9fa7d3b042fb9866: Generated new TSK 0
I20260926 20:48:03.868178 23223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:48:03.868467 23247 catalog_manager.cc:1259] Loaded cluster ID: 0815af7ea0cb470e99b59c879e0de8e2
I20260926 20:48:03.868532 23247 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8: loading cluster ID for follower catalog manager: success
I20260926 20:48:03.869304 23247 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8: acquiring CA information for follower catalog manager: success
I20260926 20:48:03.869607 23247 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f103ca7781494d03a60ccf18f7e79de8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 20:48:03.900014 23037 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:03.898259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33444 (local address 127.0.98.126:39917)
0926 20:48:03.898334 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.898336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.899303 (+ 967us) server_negotiation.cc:407] Connection header received
0926 20:48:03.899399 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.899400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.899413 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.899442 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.899933 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33444: BlockingRecv error: recv got EOF from 127.0.0.1:33444 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 20:48:03.900839 23112 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:03.898753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39694 (local address 127.0.98.125:44161)
0926 20:48:03.898794 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.898796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.900260 (+ 1464us) server_negotiation.cc:407] Connection header received
0926 20:48:03.900353 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.900353 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.900384 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.900419 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.900780 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39694: BlockingRecv error: recv got EOF from 127.0.0.1:39694 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 20:48:03.901789 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:03.899035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.0.98.124:46829)
0926 20:48:03.899078 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.899080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.901039 (+ 1959us) server_negotiation.cc:407] Connection header received
0926 20:48:03.901267 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.901268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.901280 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.901336 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.901726 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57304: BlockingRecv error: recv got EOF from 127.0.0.1:57304 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:48:03.906625 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.903271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39708 (local address 127.0.98.125:44161)
0926 20:48:03.903305 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.903307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.903502 (+ 195us) server_negotiation.cc:407] Connection header received
0926 20:48:03.903604 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.903605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.903640 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.903680 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:03.903899 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.904619 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.905698 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.905964 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.906009 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.906233 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:03.906347 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:03.906402 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.906542 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.906562 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 20:48:03.909948 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:03.908932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57316 (local address 127.0.98.124:46829)
0926 20:48:03.908966 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.908968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.909212 (+ 244us) server_negotiation.cc:407] Connection header received
0926 20:48:03.909321 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.909322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.909339 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.909391 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.909903 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57316: BlockingRecv error: recv got EOF from 127.0.0.1:57316 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:48:03.915212 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.912151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39712 (local address 127.0.98.125:44161)
0926 20:48:03.912185 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.912186 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.912404 (+ 218us) server_negotiation.cc:407] Connection header received
0926 20:48:03.912505 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.912506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.912514 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.912541 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:03.912695 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.913411 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.914456 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.914723 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.914767 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.914965 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:03.914997 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:03.915026 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.915131 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.915148 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 20:48:03.918306 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:03.917469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57328 (local address 127.0.98.124:46829)
0926 20:48:03.917519 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.917521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.917790 (+ 269us) server_negotiation.cc:407] Connection header received
0926 20:48:03.917890 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.917891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.917902 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.917932 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.918252 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57328: BlockingRecv error: recv got EOF from 127.0.0.1:57328 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 20:48:03.924566 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.921386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.0.98.125:44161)
0926 20:48:03.921419 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.921421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.921676 (+ 255us) server_negotiation.cc:407] Connection header received
0926 20:48:03.921781 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.921782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.921791 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.921820 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:03.921994 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.922680 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.923760 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.924028 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.924076 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.924280 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:03.924312 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:03.924339 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.924450 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.924467 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 20:48:03.927677 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:03.926884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57344 (local address 127.0.98.124:46829)
0926 20:48:03.926919 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.926921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.927153 (+ 232us) server_negotiation.cc:407] Connection header received
0926 20:48:03.927250 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.927251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.927262 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.927292 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.927632 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57344: BlockingRecv error: recv got EOF from 127.0.0.1:57344 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:48:03.936208 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.932826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.0.98.125:44161)
0926 20:48:03.932864 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.932866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.933102 (+ 236us) server_negotiation.cc:407] Connection header received
0926 20:48:03.933220 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.933222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.933232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.933261 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:03.933525 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.934224 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.935385 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.935680 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.935743 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.935946 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:03.935981 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:03.936010 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.936116 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.936134 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 20:48:03.939405 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:03.938518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.0.98.124:46829)
0926 20:48:03.938558 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.938560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.938816 (+ 256us) server_negotiation.cc:407] Connection header received
0926 20:48:03.938914 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.938915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.938927 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.938957 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.939351 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57358: BlockingRecv error: recv got EOF from 127.0.0.1:57358 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:48:03.952193 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.948719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39742 (local address 127.0.98.125:44161)
0926 20:48:03.948765 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.948767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.949034 (+ 267us) server_negotiation.cc:407] Connection header received
0926 20:48:03.949141 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.949142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.949152 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.949181 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:03.949360 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.950070 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.951273 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.951583 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.951624 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.951921 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:03.951955 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:03.951982 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.952087 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.952107 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 20:48:03.955405 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:03.954562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57364 (local address 127.0.98.124:46829)
0926 20:48:03.954602 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.954604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.954863 (+ 259us) server_negotiation.cc:407] Connection header received
0926 20:48:03.954963 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.954964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.954976 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.955007 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.955347 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57364: BlockingRecv error: recv got EOF from 127.0.0.1:57364 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:48:03.976454 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:03.972998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39750 (local address 127.0.98.125:44161)
0926 20:48:03.973043 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.973045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.973326 (+ 281us) server_negotiation.cc:407] Connection header received
0926 20:48:03.973433 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.973435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.973446 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.973491 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:03.973688 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.974456 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.975620 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:03.975925 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:03.975971 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:03.976194 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:03.976228 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:03.976254 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:03.976348 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 20:48:03.976369 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 20:48:03.979808 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:03.978872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.0.98.124:46829)
0926 20:48:03.978915 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:03.978917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:03.979213 (+ 296us) server_negotiation.cc:407] Connection header received
0926 20:48:03.979265 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:03.979266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:03.979278 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:03.979337 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:03.979698 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57368: BlockingRecv error: recv got EOF from 127.0.0.1:57368 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:48:04.015556 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:04.013114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39760 (local address 127.0.98.125:44161)
0926 20:48:04.013159 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:04.013161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:04.013287 (+ 126us) server_negotiation.cc:407] Connection header received
0926 20:48:04.013373 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:04.013374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:04.013385 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:04.013455 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:04.013582 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:04.014062 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:04.014841 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:04.015049 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:04.015133 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:04.015279 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:04.015318 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:04.015380 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:04.015437 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 20:48:04.015458 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 20:48:04.018195 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:04.017443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.0.98.124:46829)
0926 20:48:04.017492 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:04.017494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:04.017629 (+ 135us) server_negotiation.cc:407] Connection header received
0926 20:48:04.017702 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:04.017703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:04.017712 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:04.017758 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:04.018136 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57378: BlockingRecv error: recv got EOF from 127.0.0.1:57378 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:48:04.085909 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:04.083550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39764 (local address 127.0.98.125:44161)
0926 20:48:04.083640 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:04.083642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:04.083655 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:48:04.083784 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:04.083785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:04.083795 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:04.083846 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:04.084099 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:04.084573 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:04.085339 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:04.085522 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:04.085590 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:04.085712 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:04.085738 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:04.085775 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:04.085839 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 20:48:04.085853 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 20:48:04.088200 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:04.087642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57392 (local address 127.0.98.124:46829)
0926 20:48:04.087703 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:04.087705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:04.087773 (+ 68us) server_negotiation.cc:407] Connection header received
0926 20:48:04.087826 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:04.087827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:04.087836 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:04.087881 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:04.088135 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57392: BlockingRecv error: recv got EOF from 127.0.0.1:57392 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 20:48:04.219617 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:04.217388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39774 (local address 127.0.98.125:44161)
0926 20:48:04.217468 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:04.217470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:04.217561 (+ 91us) server_negotiation.cc:407] Connection header received
0926 20:48:04.217590 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:04.217591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:04.217602 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:04.217669 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:04.217840 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:04.218322 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:04.219068 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:04.219256 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:04.219312 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:04.219440 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:04.219468 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:04.219501 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:04.219541 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 20:48:04.219556 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 20:48:04.221993 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:04.221351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57404 (local address 127.0.98.124:46829)
0926 20:48:04.221419 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:04.221421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:04.221494 (+ 73us) server_negotiation.cc:407] Connection header received
0926 20:48:04.221563 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:04.221564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:04.221575 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:04.221630 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:04.221937 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57404: BlockingRecv error: recv got EOF from 127.0.0.1:57404 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:48:04.481632 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:04.479237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.0.98.125:44161)
0926 20:48:04.479329 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:04.479331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:04.479368 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:48:04.479398 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:04.479399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:04.479408 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:04.479454 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:04.479655 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:04.480178 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:04.480986 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:04.481175 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:04.481234 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:04.481391 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:04.481418 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:04.481457 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:04.481551 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 20:48:04.481571 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 20:48:04.484190 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:48:04.483556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57418 (local address 127.0.98.124:46829)
0926 20:48:04.483605 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:04.483608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:04.483616 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:48:04.483664 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:04.483665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:04.483677 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:04.483766 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:04.484131 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57418: BlockingRecv error: recv got EOF from 127.0.0.1:57418 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:48:04.784749 23200 catalog_manager.cc:1259] Loaded cluster ID: 0815af7ea0cb470e99b59c879e0de8e2
I20260926 20:48:04.784837 23200 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3: loading cluster ID for follower catalog manager: success
I20260926 20:48:04.785614 23200 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3: acquiring CA information for follower catalog manager: success
I20260926 20:48:04.785858 23200 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6bea3ddab804463393a0569a096e29c3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:48:05.000156 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:04.997456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39794 (local address 127.0.98.125:44161)
0926 20:48:04.997566 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:04.997568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:04.997702 (+ 134us) server_negotiation.cc:407] Connection header received
0926 20:48:04.997793 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:04.997794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:04.997806 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:04.997914 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:04.998058 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:04.998556 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:04.999341 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:04.999549 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:04.999632 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:04.999848 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:04.999883 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:04.999939 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:04.999997 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 20:48:05.000020 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":59,"threads_started":1}
I20260926 20:48:05.010341 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:05.001952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57422 (local address 127.0.98.124:46829)
0926 20:48:05.002065 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:05.002067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:05.007937 (+ 5870us) server_negotiation.cc:407] Connection header received
0926 20:48:05.008009 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:05.008010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:05.008077 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:05.008155 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:05.008287 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:05.008779 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:05.009532 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:05.009747 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:05.009828 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:05.010002 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:05.010113 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:05.010201 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:05.010224 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 20:48:05.010246 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":71,"threads_started":1}
I20260926 20:48:05.015553 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:05.012048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33446 (local address 127.0.98.126:39917)
0926 20:48:05.012144 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:05.012146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:05.012366 (+ 220us) server_negotiation.cc:407] Connection header received
0926 20:48:05.012498 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:05.012499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:05.012571 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:05.012798 (+ 227us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:05.012839 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:05.013639 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:05.014478 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:05.014785 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:05.014993 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:05.015057 (+ 64us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:05.015157 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:05.015426 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:05.015443 (+ 17us) server_negotiation.cc:299] Negotiation successful
0926 20:48:05.015468 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":54,"threads_started":1}
I20260926 20:48:05.016099 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:05.012412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57428 (local address 127.0.98.124:46829)
0926 20:48:05.012453 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:05.012455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:05.012656 (+ 201us) server_negotiation.cc:407] Connection header received
0926 20:48:05.012789 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:05.012790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:05.012802 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:05.012833 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:05.013043 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:05.013890 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:05.015186 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:05.015556 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:05.015599 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:05.015820 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:05.015853 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:05.015902 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:05.015996 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 20:48:05.016017 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:48:05.016764 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:05.012072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.0.98.125:44161)
0926 20:48:05.012131 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:05.012133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:05.012634 (+ 501us) server_negotiation.cc:407] Connection header received
0926 20:48:05.012739 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:05.012741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:05.012753 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:05.012784 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:05.012975 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:05.013671 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:05.015261 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:05.015539 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:05.015580 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:05.016142 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:05.016175 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:05.016205 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:05.016653 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 20:48:05.016677 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:48:05.021799 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 22965
I20260926 20:48:05.027544 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 23040
I20260926 20:48:05.032879 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDRmUyo/build/release/bin/kudu with pid 23115
2026-09-26T20:48:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2849 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5642 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (268414 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST